2023-12-07T07:13:39.423 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-07T07:13:39.423 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571 --verbose --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/orig.config.yaml 2023-12-07T07:13:39.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-07T07:13:39.574 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571 branch: wip-yuri8-testing-2023-12-06-1425 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7481571' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-06-1425 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: e068ebcf7ecc6503f24666fb6b152034d3fe1067 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: e068ebcf7ecc6503f24666fb6b152034d3fe1067 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-12-06-1425 sha1: e068ebcf7ecc6503f24666fb6b152034d3fe1067 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4465 sha1: e068ebcf7ecc6503f24666fb6b152034d3fe1067 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2023-12-06-1425 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e068ebcf7ecc6503f24666fb6b152034d3fe1067 targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB4RfLkzGipXJRP4cCG467MgQNHYm2xufGqX5/6siE8P+xhpBRivA6vqSff7pu0DIsxm5y8b8cm90uALQU0ZS+o= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeQXt1wvARrpJL3rnigyAmYqQUcE+qIhPN1th+Hbl2nIaVmXPX2zYdj7vZ1kMwMzNbFKgpVpWZumdUX91mLa0Q= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-07_01:25:59 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-07T07:13:39.574 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa; will attempt to use it 2023-12-07T07:13:39.575 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa/tasks 2023-12-07T07:13:39.575 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-07T07:13:39.577 INFO:teuthology.task.internal:Checking packages... 2023-12-07T07:13:39.609 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e068ebcf7ecc6503f24666fb6b152034d3fe1067' 2023-12-07T07:13:39.609 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-07T07:13:39.609 INFO:teuthology.packaging:ref: None 2023-12-07T07:13:39.609 INFO:teuthology.packaging:tag: None 2023-12-07T07:13:39.609 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-06-1425 2023-12-07T07:13:39.609 INFO:teuthology.packaging:sha1: e068ebcf7ecc6503f24666fb6b152034d3fe1067 2023-12-07T07:13:39.610 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2023-12-06-1425 2023-12-07T07:13:39.830 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-64.ge068ebcf 2023-12-07T07:13:39.830 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-07T07:13:39.837 INFO:teuthology.task.internal:no buildpackages task found 2023-12-07T07:13:39.837 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-07T07:13:39.877 INFO:teuthology.task.internal:Saving configuration 2023-12-07T07:13:39.893 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-07T07:13:39.899 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T07:13:39.924 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 07:08:17.816140', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB4RfLkzGipXJRP4cCG467MgQNHYm2xufGqX5/6siE8P+xhpBRivA6vqSff7pu0DIsxm5y8b8cm90uALQU0ZS+o='} 2023-12-07T07:13:39.944 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 07:08:17.817852', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeQXt1wvARrpJL3rnigyAmYqQUcE+qIhPN1th+Hbl2nIaVmXPX2zYdj7vZ1kMwMzNbFKgpVpWZumdUX91mLa0Q='} 2023-12-07T07:13:39.944 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-07T07:13:39.952 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-07T07:13:39.952 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-07T07:13:39.952 INFO:teuthology.run_tasks:Running task console_log... 2023-12-07T07:13:40.027 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fac1e04cee0>, signals=[15]) 2023-12-07T07:13:40.028 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-07T07:13:40.034 INFO:teuthology.task.internal:Opening connections... 2023-12-07T07:13:40.034 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T07:13:40.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:13:40.120 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T07:13:40.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:13:40.203 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-07T07:13:40.209 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2023-12-07T07:13:40.227 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-12-07T07:13:40.228 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2023-12-07T07:13:40.286 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-07T07:13:40.287 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2023-12-07T07:13:40.287 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-07T07:13:40.287 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-07T07:13:40.287 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-07T07:13:40.287 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-12-07T07:13:40.311 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2023-12-07T07:13:40.328 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2023-12-07T07:13:40.328 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2023-12-07T07:13:40.385 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-07T07:13:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-07T07:13:40.387 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-07T07:13:40.387 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2023-12-07T07:13:40.411 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-07T07:13:40.420 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-07T07:13:40.425 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-07T07:13:40.425 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2023-12-07T07:13:40.428 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2023-12-07T07:13:40.443 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-07T07:13:40.449 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-07T07:13:40.450 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2023-12-07T07:13:40.485 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2023-12-07T07:13:40.574 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-07T07:13:40.624 INFO:teuthology.run_tasks:Running task kernel... 2023-12-07T07:13:40.638 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-07T07:13:40.639 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-07T07:13:40.639 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-07T07:13:40.640 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-07T07:13:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T07:13:40.656 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2023-12-07T07:13:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T07:13:40.657 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2023-12-07T07:13:40.714 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-361.el9.x86_64 2023-12-07T07:13:40.714 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.14.0-361.el9.x86_64 2023-12-07T07:13:40.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2023-12-07T07:13:40.715 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-361.el9.x86_64 2023-12-07T07:13:40.715 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.14.0-361.el9.x86_64 2023-12-07T07:13:40.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2023-12-07T07:13:42.769 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-07T07:13:42.820 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-07T07:13:44.111 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2023-12-07T07:13:44.187 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2023-12-07T07:13:45.619 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 7.9 MB 00:01 2023-12-07T07:13:46.202 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 7.9 MB 00:01 2023-12-07T07:13:47.453 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 25 MB/s | 18 MB 00:00 2023-12-07T07:13:49.495 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 18 MB 00:02 2023-12-07T07:13:52.201 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 5.8 MB 00:01 2023-12-07T07:13:54.346 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 15 kB 00:01 2023-12-07T07:13:54.879 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 5.8 MB 00:02 2023-12-07T07:13:56.446 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-12-07T07:13:57.064 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 20 MB 00:02 2023-12-07T07:13:58.581 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 20 MB 00:02 2023-12-07T07:14:01.756 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-07T07:14:03.389 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2023-12-07T07:14:03.987 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-07T07:14:03.987 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-07T07:14:04.079 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:14:04.092 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:14:04.093 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:14:04.093 INFO:teuthology.orchestra.run.smithi097.stdout:Install 4 Packages 2023-12-07T07:14:04.093 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:14:04.093 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 96 M 2023-12-07T07:14:04.093 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 121 M 2023-12-07T07:14:04.093 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:14:05.424 INFO:teuthology.orchestra.run.smithi097.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 5.3 MB/s | 5.6 MB 00:01 2023-12-07T07:14:05.621 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-07T07:14:05.621 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-07T07:14:05.710 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 5.14.0-390.el9 baseos 5.6 M 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 5.14.0-390.el9 baseos 20 M 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 5.14.0-390.el9 baseos 38 M 2023-12-07T07:14:05.715 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core x86_64 5.14.0-390.el9 baseos 32 M 2023-12-07T07:14:05.716 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:14:05.716 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:14:05.716 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:14:05.716 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2023-12-07T07:14:05.716 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:14:05.716 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 96 M 2023-12-07T07:14:05.716 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 121 M 2023-12-07T07:14:05.717 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:14:06.216 INFO:teuthology.orchestra.run.smithi097.stdout:(2/4): kernel-core-5.14.0-390.el9.x86_64.rpm 11 MB/s | 20 MB 00:01 2023-12-07T07:14:07.191 INFO:teuthology.orchestra.run.smithi097.stdout:(3/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 13 MB/s | 38 MB 00:02 2023-12-07T07:14:07.482 INFO:teuthology.orchestra.run.smithi097.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 16 MB/s | 32 MB 00:02 2023-12-07T07:14:07.485 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:14:07.486 INFO:teuthology.orchestra.run.smithi097.stdout:Total 28 MB/s | 96 MB 00:03 2023-12-07T07:14:07.590 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): kernel-5.14.0-390.el9.x86_64.rpm 3.5 MB/s | 5.6 MB 00:01 2023-12-07T07:14:08.344 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:14:08.712 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:14:08.712 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:14:09.157 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): kernel-modules-5.14.0-390.el9.x86_64.rpm 12 MB/s | 38 MB 00:03 2023-12-07T07:14:10.026 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:14:10.026 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:14:10.032 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): kernel-core-5.14.0-390.el9.x86_64.rpm 5.0 MB/s | 20 MB 00:04 2023-12-07T07:14:12.669 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:14:13.294 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-07T07:14:13.433 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T07:14:16.167 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T07:14:16.398 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T07:14:20.974 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T07:14:21.160 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:25.539 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): kernel-modules-core-5.14.0-390.el9.x86_6 1.8 MB/s | 32 MB 00:17 2023-12-07T07:14:25.566 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:14:25.567 INFO:teuthology.orchestra.run.smithi154.stdout:Total 4.9 MB/s | 96 MB 00:19 2023-12-07T07:14:25.809 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:26.393 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:14:26.771 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:14:26.772 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:14:28.098 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:14:28.098 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:14:31.010 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:14:31.650 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-core-5.14.0-390.el9.x86_64 1/4 2023-12-07T07:14:31.781 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T07:14:34.575 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T07:14:34.761 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T07:14:39.364 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T07:14:39.568 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:44.162 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:54.716 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:54.758 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:55.146 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:55.146 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-07T07:14:55.146 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T07:14:55.147 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:14:55.533 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:14:56.025 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T07:14:56.163 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T07:14:56.163 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T07:14:56.163 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-07T07:14:56.163 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-07T07:14:56.163 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-07T07:14:56.163 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-07T07:14:56.164 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2023-12-07T07:14:56.164 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T07:14:56.282 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T07:14:56.282 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T07:14:56.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2023-12-07T07:14:56.322 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2023-12-07T07:14:56.322 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2023-12-07T07:14:56.322 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2023-12-07T07:14:56.323 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2023-12-07T07:14:56.324 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:14:56.324 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-07T07:14:56.324 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-07T07:14:56.324 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-07T07:14:56.324 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2023-12-07T07:14:56.324 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:14:56.324 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2023-12-07T07:14:56.324 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-07T07:14:56.324 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 5.14.0-390.el9.x86_64 2023-12-07T07:14:56.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-07T07:14:56.990 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2023-12-07T07:14:58.336 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-07T07:14:58.381 INFO:teuthology.orchestra.run.smithi097.stderr:done 2023-12-07T07:14:58.383 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2023-12-07T07:14:58.399 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.Ga2TJClh6M 2023-12-07T07:14:58.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ga2TJClh6M 2023-12-07T07:14:58.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.Ga2TJClh6M 2023-12-07T07:14:58.607 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.Ga2TJClh6M is 6KB 2023-12-07T07:14:58.655 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.Ga2TJClh6M 2023-12-07T07:14:58.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2023-12-07T07:14:58.739 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-07T07:14:58.739 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-07T07:14:58.739 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-07T07:14:58.739 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-07T07:14:58.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-07T07:14:58.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2023-12-07T07:15:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:15:12.875 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:15:13.339 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:15:13.339 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-5.14.0-390.el9.x86_64 1/4 2023-12-07T07:15:13.339 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-5.14.0-390.el9.x86_64 2/4 2023-12-07T07:15:13.339 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-5.14.0-390.el9.x86_64 3/4 2023-12-07T07:15:13.759 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-core-5.14.0-390.el9.x86_64 4/4 2023-12-07T07:15:13.759 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:15:13.759 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:15:13.760 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-5.14.0-390.el9.x86_64 2023-12-07T07:15:13.760 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core-5.14.0-390.el9.x86_64 2023-12-07T07:15:13.760 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-5.14.0-390.el9.x86_64 2023-12-07T07:15:13.760 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core-5.14.0-390.el9.x86_64 2023-12-07T07:15:13.760 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:15:13.760 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:15:14.242 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T07:15:14.393 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T07:15:14.393 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T07:15:14.393 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-390.el9.x86_64 2023-12-07T07:15:14.393 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-07T07:15:14.393 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-07T07:15:14.394 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-07T07:15:14.394 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2023-12-07T07:15:14.394 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2023-12-07T07:15:14.510 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-390.el9.x86_64 2023-12-07T07:15:14.511 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-390.el9.x86_64 2023-12-07T07:15:14.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.06 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 68.el9 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Group : Unspecified 2023-12-07T07:15:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 8264254 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : builder@centos.org 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2023-12-07T07:15:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2023-12-07T07:15:14.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:15:14.555 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2023-12-07T07:15:14.556 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-07T07:15:14.556 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 5.14.0-390.el9.x86_64 2023-12-07T07:15:14.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-07T07:15:15.128 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2023-12-07T07:15:16.516 INFO:teuthology.orchestra.run.smithi154.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-07T07:15:16.555 INFO:teuthology.orchestra.run.smithi154.stderr:done 2023-12-07T07:15:16.557 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2023-12-07T07:15:16.572 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.3CWCmdil1N 2023-12-07T07:15:16.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3CWCmdil1N 2023-12-07T07:15:16.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.3CWCmdil1N 2023-12-07T07:15:16.783 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.3CWCmdil1N is 6KB 2023-12-07T07:15:16.832 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.3CWCmdil1N 2023-12-07T07:15:16.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2023-12-07T07:15:16.918 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-07T07:15:16.918 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-07T07:15:16.919 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-07T07:15:16.919 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64.conf 2023-12-07T07:15:16.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-390.el9.x86_64 2023-12-07T07:15:17.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2023-12-07T07:15:28.990 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-07T07:15:28.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-07T07:15:28.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:15:47.070 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-07T07:15:47.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-12-07T07:15:47.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:15:47.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-12-07T07:15:56.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-07T07:15:56.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:15:59.490 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-12-07T07:16:05.603 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-12-07T07:16:11.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-07T07:16:11.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:16:14.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-12-07T07:16:14.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:16:17.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-12-07T07:16:29.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-12-07T07:16:29.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:16:43.383 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-12-07T07:16:43.723 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-07T07:16:43.723 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-390.el9.x86_64"... 2023-12-07T07:16:43.724 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2023-12-07T07:16:43.738 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-390.el9.x86_64 2023-12-07T07:16:43.739 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-07T07:16:43.739 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-07T07:16:43.740 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-07T07:16:44.741 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-07T07:16:44.741 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-07T07:16:44.834 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2023-12-07T07:16:44.870 DEBUG:teuthology.parallel:result is None 2023-12-07T07:16:48.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-12-07T07:17:03.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-12-07T07:17:03.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:17:05.886 DEBUG:teuthology.orchestra.run.smithi154:> true 2023-12-07T07:17:06.368 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-12-07T07:17:06.369 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-390.el9.x86_64"... 2023-12-07T07:17:06.369 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2023-12-07T07:17:06.383 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-390.el9.x86_64 2023-12-07T07:17:06.383 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-390.el9.x86_64 vs 5.14.0-390.el9.x86_64 2023-12-07T07:17:06.383 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-07T07:17:06.383 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-07T07:17:07.383 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-07T07:17:07.384 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-07T07:17:07.470 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2023-12-07T07:17:07.506 DEBUG:teuthology.parallel:result is None 2023-12-07T07:17:07.507 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-07T07:17:07.516 INFO:teuthology.task.internal:Creating test directory... 2023-12-07T07:17:07.517 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-07T07:17:07.520 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-07T07:17:07.537 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-07T07:17:07.544 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-07T07:17:07.551 INFO:teuthology.task.internal:Creating archive directory... 2023-12-07T07:17:07.551 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-07T07:17:07.580 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-07T07:17:07.627 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-07T07:17:07.634 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-07T07:17:07.634 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-07T07:17:07.659 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-07T07:17:07.697 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T07:17:07.701 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T07:17:07.711 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T07:17:07.714 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T07:17:07.715 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-07T07:17:07.746 INFO:teuthology.task.internal:Configuring sudo... 2023-12-07T07:17:07.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-07T07:17:07.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-07T07:17:07.782 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-07T07:17:07.793 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-07T07:17:07.793 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-07T07:17:07.825 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-07T07:17:07.839 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T07:17:07.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T07:17:07.978 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T07:17:08.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T07:17:08.103 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:17:08.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-07T07:17:08.171 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T07:17:08.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T07:17:08.274 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T07:17:08.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T07:17:08.359 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:17:08.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-07T07:17:08.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2023-12-07T07:17:08.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2023-12-07T07:17:08.475 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-07T07:17:08.515 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-07T07:17:08.753 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-07T07:17:08.761 INFO:teuthology.task.internal:Starting timer... 2023-12-07T07:17:08.761 INFO:teuthology.run_tasks:Running task pcp... 2023-12-07T07:17:08.774 INFO:teuthology.run_tasks:Running task selinux... 2023-12-07T07:17:08.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2023-12-07T07:17:08.851 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2023-12-07T07:17:08.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2023-12-07T07:17:08.928 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2023-12-07T07:17:08.929 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-07T07:17:08.930 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2023-12-07T07:17:08.951 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2023-12-07T07:17:08.951 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2023-12-07T07:17:08.969 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2023-12-07T07:17:08.970 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2023-12-07T07:17:08.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-07T07:17:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T07:17:09.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-07T07:17:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T07:17:09.048 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-07T07:17:09.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2023-12-07T07:17:09.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2023-12-07T07:17:09.138 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-07T07:17:09.148 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-07T07:17:09.148 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-07T07:17:09.165 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-07T07:17:09.166 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-07T07:22:53.436 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2023-12-07T07:22:53.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-07T07:22:53.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:22:53.525 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-12-07T07:22:53.597 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-07T07:22:53.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-12-07T07:22:53.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T07:22:53.680 DEBUG:teuthology.orchestra.run.smithi154:> true 2023-12-07T07:22:53.749 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-12-07T07:22:53.749 INFO:teuthology.run_tasks:Running task clock... 2023-12-07T07:22:53.765 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-07T07:22:53.765 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-07T07:22:53.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T07:22:53.768 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-07T07:22:53.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T07:22:53.798 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-07T07:22:53.815 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-07T07:22:53.819 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-07T07:22:53.836 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-07T07:22:53.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2023-12-07T07:22:53.858 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2023-12-07T07:22:53.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2023-12-07T07:22:53.874 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-07T07:22:53.879 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2023-12-07T07:22:53.891 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-07T07:22:53.896 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-07T07:22:53.913 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-07T07:22:53.933 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2023-12-07T07:22:53.937 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-07T07:22:53.937 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2023-12-07T07:22:53.937 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.937 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.937 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.937 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.956 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2023-12-07T07:22:53.959 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-07T07:22:53.959 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2023-12-07T07:22:53.959 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.960 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.960 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.960 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T07:22:53.961 INFO:teuthology.run_tasks:Running task install... 2023-12-07T07:22:53.975 DEBUG:teuthology.task.install:project ceph 2023-12-07T07:22:53.975 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067'}} 2023-12-07T07:22:53.976 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067'} 2023-12-07T07:22:53.976 INFO:teuthology.task.install:Using flavor: default 2023-12-07T07:22:53.984 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-12-07T07:22:53.984 INFO:teuthology.task.install:extra packages: [] 2023-12-07T07:22:53.985 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067', 'tag': None, 'wait_for_package': False} 2023-12-07T07:22:53.985 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e068ebcf7ecc6503f24666fb6b152034d3fe1067 2023-12-07T07:22:53.991 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067', 'tag': None, 'wait_for_package': False} 2023-12-07T07:22:53.991 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e068ebcf7ecc6503f24666fb6b152034d3fe1067 2023-12-07T07:22:54.218 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/ 2023-12-07T07:22:54.218 INFO:teuthology.task.install.rpm:Package version is 19.0.0-64.ge068ebcf 2023-12-07T07:22:54.226 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/ 2023-12-07T07:22:54.226 INFO:teuthology.task.install.rpm:Package version is 19.0.0-64.ge068ebcf 2023-12-07T07:22:54.338 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-07T07:22:54.338 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:22:54.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-07T07:22:54.366 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-07T07:22:54.367 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e068ebcf7ecc6503f24666fb6b152034d3fe1067/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-07T07:22:54.387 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-06-1425/e068ebcf7ecc6503f24666fb6b152034d3fe1067/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-07T07:22:54.387 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:22:54.387 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-07T07:22:54.415 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-07T07:22:54.415 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e068ebcf7ecc6503f24666fb6b152034d3fe1067/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-07T07:22:54.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-07T07:22:54.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-07T07:22:54.514 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-07T07:22:54.563 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-07T07:22:54.579 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2023-12-07T07:22:54.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2023-12-07T07:22:54.629 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2023-12-07T07:22:54.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2023-12-07T07:22:54.821 INFO:teuthology.orchestra.run.smithi097.stdout:73 files removed 2023-12-07T07:22:54.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw 2023-12-07T07:22:54.874 INFO:teuthology.orchestra.run.smithi154.stdout:73 files removed 2023-12-07T07:22:54.902 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2023-12-07T07:22:55.560 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 196 kB/s | 78 kB 00:00 2023-12-07T07:22:55.711 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 160 kB/s | 78 kB 00:00 2023-12-07T07:22:55.944 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 28 kB/s | 9.5 kB 00:00 2023-12-07T07:22:56.071 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 31 kB/s | 9.5 kB 00:00 2023-12-07T07:22:56.320 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2023-12-07T07:22:56.471 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2023-12-07T07:22:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-07T07:22:57.090 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-12-07T07:22:58.214 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2023-12-07T07:22:58.591 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2023-12-07T07:22:59.980 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-12-07T07:23:00.358 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-12-07T07:23:01.790 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 25 MB/s | 18 MB 00:00 2023-12-07T07:23:02.307 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-12-07T07:23:05.732 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.8 MB 00:00 2023-12-07T07:23:07.050 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.8 MB 00:01 2023-12-07T07:23:07.533 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2023-12-07T07:23:08.534 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2023-12-07T07:23:13.229 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 20 MB 00:04 2023-12-07T07:23:17.910 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2023-12-07T07:23:20.467 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:23:20.486 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:23:20.486 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:23:20.486 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:23:20.486 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:23:20.486 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 9.7 M 2023-12-07T07:23:20.486 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2023-12-07T07:23:20.486 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 3.3 M 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 3.0 M 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.2 M 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 20 M 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 24 k 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 722 k 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-07T07:23:20.487 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 484 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.1 M 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 45 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 129 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 161 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 320 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 298 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 99 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-07T07:23:20.488 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout:Install 27 Packages 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 57 M 2023-12-07T07:23:20.489 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:23:21.686 INFO:teuthology.orchestra.run.smithi154.stdout:(1/29): ceph-base-19.0.0-64.ge068ebcf.el9.x86_6 6.3 MB/s | 5.2 MB 00:00 2023-12-07T07:23:21.736 INFO:teuthology.orchestra.run.smithi154.stdout:(2/29): ceph-selinux-19.0.0-64.ge068ebcf.el9.x8 482 kB/s | 24 kB 00:00 2023-12-07T07:23:21.869 INFO:teuthology.orchestra.run.smithi154.stdout:(3/29): libcephfs2-19.0.0-64.ge068ebcf.el9.x86_ 5.3 MB/s | 722 kB 00:00 2023-12-07T07:23:21.986 INFO:teuthology.orchestra.run.smithi154.stdout:(4/29): libradosstriper1-19.0.0-64.ge068ebcf.el 4.1 MB/s | 484 kB 00:00 2023-12-07T07:23:22.162 INFO:teuthology.orchestra.run.smithi154.stdout:(5/29): ceph-radosgw-19.0.0-64.ge068ebcf.el9.x8 7.4 MB/s | 9.7 MB 00:01 2023-12-07T07:23:22.229 INFO:teuthology.orchestra.run.smithi154.stdout:(6/29): python3-ceph-argparse-19.0.0-64.ge068eb 679 kB/s | 45 kB 00:00 2023-12-07T07:23:22.306 INFO:teuthology.orchestra.run.smithi154.stdout:(7/29): python3-ceph-common-19.0.0-64.ge068ebcf 1.6 MB/s | 129 kB 00:00 2023-12-07T07:23:22.389 INFO:teuthology.orchestra.run.smithi154.stdout:(8/29): python3-cephfs-19.0.0-64.ge068ebcf.el9. 1.9 MB/s | 161 kB 00:00 2023-12-07T07:23:22.481 INFO:teuthology.orchestra.run.smithi154.stdout:(9/29): python3-rados-19.0.0-64.ge068ebcf.el9.x 3.4 MB/s | 320 kB 00:00 2023-12-07T07:23:22.574 INFO:teuthology.orchestra.run.smithi154.stdout:(10/29): librgw2-19.0.0-64.ge068ebcf.el9.x86_64 8.6 MB/s | 5.1 MB 00:00 2023-12-07T07:23:22.599 INFO:teuthology.orchestra.run.smithi154.stdout:(11/29): python3-rbd-19.0.0-64.ge068ebcf.el9.x8 2.5 MB/s | 298 kB 00:00 2023-12-07T07:23:22.624 INFO:teuthology.orchestra.run.smithi154.stdout:(12/29): python3-rgw-19.0.0-64.ge068ebcf.el9.x8 1.9 MB/s | 99 kB 00:00 2023-12-07T07:23:22.825 INFO:teuthology.orchestra.run.smithi154.stdout:(13/29): ceph-common-19.0.0-64.ge068ebcf.el9.x8 9.9 MB/s | 20 MB 00:01 2023-12-07T07:23:22.967 INFO:teuthology.orchestra.run.smithi154.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 288 kB/s | 106 kB 00:00 2023-12-07T07:23:22.992 INFO:teuthology.orchestra.run.smithi154.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 433 kB/s | 159 kB 00:00 2023-12-07T07:23:23.017 INFO:teuthology.orchestra.run.smithi154.stdout:(16/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 309 kB/s | 44 kB 00:00 2023-12-07T07:23:23.059 INFO:teuthology.orchestra.run.smithi154.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2023-12-07T07:23:23.085 INFO:teuthology.orchestra.run.smithi154.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 621 kB/s | 41 kB 00:00 2023-12-07T07:23:23.110 INFO:teuthology.orchestra.run.smithi154.stdout:(19/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2023-12-07T07:23:23.252 INFO:teuthology.orchestra.run.smithi154.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2023-12-07T07:23:23.360 INFO:teuthology.orchestra.run.smithi154.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2023-12-07T07:23:23.385 INFO:teuthology.orchestra.run.smithi154.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 947 kB/s | 309 kB 00:00 2023-12-07T07:23:23.444 INFO:teuthology.orchestra.run.smithi154.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2023-12-07T07:23:23.511 INFO:teuthology.orchestra.run.smithi154.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2023-12-07T07:23:23.545 INFO:teuthology.orchestra.run.smithi154.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2023-12-07T07:23:23.595 INFO:teuthology.orchestra.run.smithi154.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2023-12-07T07:23:23.620 INFO:teuthology.orchestra.run.smithi154.stdout:(27/29): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2023-12-07T07:23:23.870 INFO:teuthology.orchestra.run.smithi154.stdout:(28/29): librados2-19.0.0-64.ge068ebcf.el9.x86_ 10 MB/s | 3.3 MB 00:00 2023-12-07T07:23:23.929 INFO:teuthology.orchestra.run.smithi154.stdout:(29/29): librbd1-19.0.0-64.ge068ebcf.el9.x86_64 9.1 MB/s | 3.0 MB 00:00 2023-12-07T07:23:23.932 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:23:23.933 INFO:teuthology.orchestra.run.smithi154.stdout:Total 16 MB/s | 57 MB 00:03 2023-12-07T07:23:23.933 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:23:23.965 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:23:23.965 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:23:24.434 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:23:24.435 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:23:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:23:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-07T07:23:25.892 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-07T07:23:25.943 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-07T07:23:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-07T07:23:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 5/31 2023-12-07T07:23:26.416 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 5/31 2023-12-07T07:23:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 6/31 2023-12-07T07:23:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 7/31 2023-12-07T07:23:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 7/31 2023-12-07T07:23:26.801 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-07T07:23:26.861 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.0.0-64.ge068ebcf.el9.x8 9/31 2023-12-07T07:23:26.929 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 10/31 2023-12-07T07:23:26.954 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 11/31 2023-12-07T07:23:27.019 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 11/31 2023-12-07T07:23:27.122 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-07T07:23:27.156 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-07T07:23:27.197 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-07T07:23:27.229 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-07T07:23:27.266 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-07T07:23:27.299 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-07T07:23:27.600 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-07T07:23:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 19/31 2023-12-07T07:23:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 19/31 2023-12-07T07:23:27.734 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 20/31 2023-12-07T07:23:27.770 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-07T07:23:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-07T07:23:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 23/31 2023-12-07T07:23:28.311 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 23/31 2023-12-07T07:23:28.379 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 24/31 2023-12-07T07:23:28.409 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_ 25/31 2023-12-07T07:23:28.543 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 965 kB/s | 20 MB 00:20 2023-12-07T07:23:30.629 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 26/31 2023-12-07T07:23:30.657 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 26/31 2023-12-07T07:23:31.245 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 26/31 2023-12-07T07:23:31.277 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 27/31 2023-12-07T07:23:31.368 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 27/31 2023-12-07T07:23:31.378 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-07T07:23:31.378 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-07T07:23:31.378 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 28/31 2023-12-07T07:23:33.274 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2023-12-07T07:23:36.048 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:23:36.055 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 9.7 M 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 3.3 M 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 3.0 M 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.2 M 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 20 M 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 24 k 2023-12-07T07:23:36.056 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 722 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 484 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.1 M 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 45 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 129 k 2023-12-07T07:23:36.057 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 161 k 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 320 k 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 298 k 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 99 k 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout:Install 27 Packages 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 2 Packages 2023-12-07T07:23:36.058 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:23:36.059 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 57 M 2023-12-07T07:23:36.059 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:23:36.759 INFO:teuthology.orchestra.run.smithi097.stdout:(1/29): ceph-base-19.0.0-64.ge068ebcf.el9.x86_6 10 MB/s | 5.2 MB 00:00 2023-12-07T07:23:36.802 INFO:teuthology.orchestra.run.smithi097.stdout:(2/29): ceph-selinux-19.0.0-64.ge068ebcf.el9.x8 579 kB/s | 24 kB 00:00 2023-12-07T07:23:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:(3/29): libcephfs2-19.0.0-64.ge068ebcf.el9.x86_ 11 MB/s | 722 kB 00:00 2023-12-07T07:23:36.917 INFO:teuthology.orchestra.run.smithi097.stdout:(4/29): libradosstriper1-19.0.0-64.ge068ebcf.el 9.5 MB/s | 484 kB 00:00 2023-12-07T07:23:37.218 INFO:teuthology.orchestra.run.smithi097.stdout:(5/29): librgw2-19.0.0-64.ge068ebcf.el9.x86_64. 17 MB/s | 5.1 MB 00:00 2023-12-07T07:23:37.251 INFO:teuthology.orchestra.run.smithi097.stdout:(6/29): python3-ceph-argparse-19.0.0-64.ge068eb 1.3 MB/s | 45 kB 00:00 2023-12-07T07:23:37.302 INFO:teuthology.orchestra.run.smithi097.stdout:(7/29): python3-ceph-common-19.0.0-64.ge068ebcf 2.5 MB/s | 129 kB 00:00 2023-12-07T07:23:37.360 INFO:teuthology.orchestra.run.smithi097.stdout:(8/29): python3-cephfs-19.0.0-64.ge068ebcf.el9. 2.7 MB/s | 161 kB 00:00 2023-12-07T07:23:37.436 INFO:teuthology.orchestra.run.smithi097.stdout:(9/29): python3-rados-19.0.0-64.ge068ebcf.el9.x 4.2 MB/s | 320 kB 00:00 2023-12-07T07:23:37.502 INFO:teuthology.orchestra.run.smithi097.stdout:(10/29): python3-rbd-19.0.0-64.ge068ebcf.el9.x8 4.4 MB/s | 298 kB 00:00 2023-12-07T07:23:37.544 INFO:teuthology.orchestra.run.smithi097.stdout:(11/29): python3-rgw-19.0.0-64.ge068ebcf.el9.x8 2.3 MB/s | 99 kB 00:00 2023-12-07T07:23:37.653 INFO:teuthology.orchestra.run.smithi097.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2023-12-07T07:23:37.703 INFO:teuthology.orchestra.run.smithi097.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-12-07T07:23:37.745 INFO:teuthology.orchestra.run.smithi097.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2023-12-07T07:23:37.896 INFO:teuthology.orchestra.run.smithi097.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2023-12-07T07:23:37.962 INFO:teuthology.orchestra.run.smithi097.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-12-07T07:23:38.021 INFO:teuthology.orchestra.run.smithi097.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 705 kB/s | 41 kB 00:00 2023-12-07T07:23:38.146 INFO:teuthology.orchestra.run.smithi097.stdout:(18/29): ceph-radosgw-19.0.0-64.ge068ebcf.el9.x 5.1 MB/s | 9.7 MB 00:01 2023-12-07T07:23:38.758 INFO:teuthology.orchestra.run.smithi097.stdout:(19/29): ceph-common-19.0.0-64.ge068ebcf.el9.x8 7.8 MB/s | 20 MB 00:02 2023-12-07T07:23:39.033 INFO:teuthology.orchestra.run.smithi097.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 305 kB/s | 309 kB 00:01 2023-12-07T07:23:39.242 INFO:teuthology.orchestra.run.smithi097.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2023-12-07T07:23:39.267 INFO:teuthology.orchestra.run.smithi097.stdout:(22/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 51 kB/s | 26 kB 00:00 2023-12-07T07:23:39.385 INFO:teuthology.orchestra.run.smithi097.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 473 kB/s | 67 kB 00:00 2023-12-07T07:23:39.544 INFO:teuthology.orchestra.run.smithi097.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2023-12-07T07:23:39.568 INFO:teuthology.orchestra.run.smithi097.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2023-12-07T07:23:39.777 INFO:teuthology.orchestra.run.smithi097.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2023-12-07T07:23:39.803 INFO:teuthology.orchestra.run.smithi097.stdout:(27/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2023-12-07T07:23:39.861 INFO:teuthology.orchestra.run.smithi097.stdout:(28/29): librados2-19.0.0-64.ge068ebcf.el9.x86_ 11 MB/s | 3.3 MB 00:00 2023-12-07T07:23:40.128 INFO:teuthology.orchestra.run.smithi097.stdout:(29/29): librbd1-19.0.0-64.ge068ebcf.el9.x86_64 8.7 MB/s | 3.0 MB 00:00 2023-12-07T07:23:40.133 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:23:40.133 INFO:teuthology.orchestra.run.smithi097.stdout:Total 14 MB/s | 57 MB 00:04 2023-12-07T07:23:40.134 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:23:40.165 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:23:40.165 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:23:40.636 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:23:40.636 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 28/31 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2023-12-07T07:23:40.912 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:41.793 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 29/31 2023-12-07T07:23:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 29/31 2023-12-07T07:23:41.815 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:23:41.815 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-07T07:23:41.815 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T07:23:41.815 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T07:23:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T07:23:41.904 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T07:23:41.904 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T07:23:42.063 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:23:42.152 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-07T07:23:42.183 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-07T07:23:42.241 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-07T07:23:42.616 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-07T07:23:42.648 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 5/31 2023-12-07T07:23:42.723 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 5/31 2023-12-07T07:23:42.834 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 6/31 2023-12-07T07:23:42.866 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 7/31 2023-12-07T07:23:42.920 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 7/31 2023-12-07T07:23:42.949 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-07T07:23:42.998 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-64.ge068ebcf.el9.x8 9/31 2023-12-07T07:23:43.076 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 10/31 2023-12-07T07:23:43.087 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T07:23:43.087 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 1/31 2023-12-07T07:23:43.087 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 2/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 3/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 4/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 5/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 6/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 7/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.0.0-64.ge068ebcf.el9.x8 8/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_ 9/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 10/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 11/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 12/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 13/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-07T07:23:43.088 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 28/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-07T07:23:43.089 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 30/31 2023-12-07T07:23:43.102 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 11/31 2023-12-07T07:23:43.167 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 11/31 2023-12-07T07:23:43.273 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-07T07:23:43.302 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-07T07:23:43.343 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-07T07:23:43.374 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-07T07:23:43.403 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-07T07:23:43.436 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-07T07:23:43.731 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-07T07:23:43.761 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 19/31 2023-12-07T07:23:43.841 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 19/31 2023-12-07T07:23:43.874 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 20/31 2023-12-07T07:23:43.909 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-07T07:23:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T07:23:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:44.255 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-07T07:23:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-3.el9.x86_64 2023-12-07T07:23:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-07T07:23:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:44.258 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:23:44.355 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-07T07:23:44.380 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 23/31 2023-12-07T07:23:44.434 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 23/31 2023-12-07T07:23:44.503 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 24/31 2023-12-07T07:23:44.533 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_ 25/31 2023-12-07T07:23:44.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2023-12-07T07:23:45.211 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:28 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:23:45.356 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:23:45.356 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:23:45.356 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:23:45.356 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 43 M 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 164 k 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2023-12-07T07:23:45.357 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:45.358 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 43 M 2023-12-07T07:23:45.358 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 176 M 2023-12-07T07:23:45.358 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:23:45.576 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 860 kB/s | 186 kB 00:00 2023-12-07T07:23:45.634 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2023-12-07T07:23:45.709 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2023-12-07T07:23:45.768 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2023-12-07T07:23:45.835 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): libcephsqlite-19.0.0-64.ge068ebcf.el9.x8 344 kB/s | 164 kB 00:00 2023-12-07T07:23:46.667 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 26/31 2023-12-07T07:23:46.704 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 26/31 2023-12-07T07:23:47.273 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 26/31 2023-12-07T07:23:47.308 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 27/31 2023-12-07T07:23:47.391 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 27/31 2023-12-07T07:23:47.391 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-07T07:23:47.392 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-07T07:23:47.392 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:23:47.426 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 28/31 2023-12-07T07:23:47.544 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): ceph-test-19.0.0-64.ge068ebcf.el9.x86_64 19 MB/s | 43 MB 00:02 2023-12-07T07:23:47.547 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:23:47.547 INFO:teuthology.orchestra.run.smithi154.stdout:Total 20 MB/s | 43 MB 00:02 2023-12-07T07:23:47.547 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:23:47.568 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:23:47.568 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:23:47.917 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:23:47.917 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:23:48.465 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:23:48.549 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-07T07:23:48.586 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-07T07:23:48.618 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-07T07:23:48.667 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-07T07:23:48.697 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 5/6 2023-12-07T07:23:52.668 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 5/6 2023-12-07T07:23:52.745 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 6/6 2023-12-07T07:23:54.886 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 6/6 2023-12-07T07:23:54.887 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 1/6 2023-12-07T07:23:54.887 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 2/6 2023-12-07T07:23:54.887 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-07T07:23:54.887 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-07T07:23:54.887 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-07T07:23:55.255 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-07T07:23:55.255 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:55.255 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:23:55.255 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 jq-1.6-16.el9.x86_64 2023-12-07T07:23:55.255 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-12-07T07:23:55.256 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-12-07T07:23:55.256 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:55.256 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:23:55.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2023-12-07T07:23:56.134 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:39 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:23:56.282 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:23:56.284 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2023-12-07T07:23:56.284 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:23:56.284 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2023-12-07T07:23:56.284 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:23:56.284 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 6.5 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 2.1 M 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.5 M 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 243 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 4.4 M 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 18 M 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-07T07:23:56.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-07T07:23:56.286 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout:Install 36 Packages 2023-12-07T07:23:56.287 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:23:56.288 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 30 M 2023-12-07T07:23:56.288 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 111 M 2023-12-07T07:23:56.288 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:23:56.629 INFO:teuthology.orchestra.run.smithi154.stdout:(1/36): ceph-19.0.0-64.ge068ebcf.el9.x86_64.rpm 40 kB/s | 6.5 kB 00:00 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 28/31 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2023-12-07T07:23:56.640 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:23:57.045 INFO:teuthology.orchestra.run.smithi154.stdout:(2/36): ceph-mon-19.0.0-64.ge068ebcf.el9.x86_64 11 MB/s | 4.4 MB 00:00 2023-12-07T07:23:57.188 INFO:teuthology.orchestra.run.smithi154.stdout:(3/36): ceph-mds-19.0.0-64.ge068ebcf.el9.x86_64 2.9 MB/s | 2.1 MB 00:00 2023-12-07T07:23:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:(4/36): ceph-mgr-modules-core-19.0.0-64.ge068eb 2.2 MB/s | 243 kB 00:00 2023-12-07T07:23:57.396 INFO:teuthology.orchestra.run.smithi154.stdout:(5/36): ceph-mgr-19.0.0-64.ge068ebcf.el9.x86_64 1.6 MB/s | 1.5 MB 00:00 2023-12-07T07:23:57.463 INFO:teuthology.orchestra.run.smithi154.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-12-07T07:23:57.488 INFO:teuthology.orchestra.run.smithi154.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2023-12-07T07:23:57.508 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 29/31 2023-12-07T07:23:57.514 INFO:teuthology.orchestra.run.smithi154.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2023-12-07T07:23:57.531 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 29/31 2023-12-07T07:23:57.531 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:23:57.531 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-07T07:23:57.532 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T07:23:57.532 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-07T07:23:57.532 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:23:57.533 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T07:23:57.539 INFO:teuthology.orchestra.run.smithi154.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2023-12-07T07:23:57.564 INFO:teuthology.orchestra.run.smithi154.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2023-12-07T07:23:57.589 INFO:teuthology.orchestra.run.smithi154.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-07T07:23:57.635 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-07T07:23:57.635 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T07:23:57.648 INFO:teuthology.orchestra.run.smithi154.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2023-12-07T07:23:57.673 INFO:teuthology.orchestra.run.smithi154.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-12-07T07:23:57.707 INFO:teuthology.orchestra.run.smithi154.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2023-12-07T07:23:57.724 INFO:teuthology.orchestra.run.smithi154.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-07T07:23:57.749 INFO:teuthology.orchestra.run.smithi154.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2023-12-07T07:23:57.816 INFO:teuthology.orchestra.run.smithi154.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2023-12-07T07:23:57.991 INFO:teuthology.orchestra.run.smithi154.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 625 kB/s | 151 kB 00:00 2023-12-07T07:23:58.017 INFO:teuthology.orchestra.run.smithi154.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 217 kB/s | 43 kB 00:00 2023-12-07T07:23:58.201 INFO:teuthology.orchestra.run.smithi154.stdout:(20/36): ceph-osd-19.0.0-64.ge068ebcf.el9.x86_6 15 MB/s | 18 MB 00:01 2023-12-07T07:23:58.217 INFO:teuthology.orchestra.run.smithi154.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2023-12-07T07:23:58.301 INFO:teuthology.orchestra.run.smithi154.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2023-12-07T07:23:58.318 INFO:teuthology.orchestra.run.smithi154.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2023-12-07T07:23:58.336 INFO:teuthology.orchestra.run.smithi154.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2023-12-07T07:23:58.384 INFO:teuthology.orchestra.run.smithi154.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 466 kB/s | 23 kB 00:00 2023-12-07T07:23:58.401 INFO:teuthology.orchestra.run.smithi154.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2023-12-07T07:23:58.418 INFO:teuthology.orchestra.run.smithi154.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2023-12-07T07:23:58.435 INFO:teuthology.orchestra.run.smithi154.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 912 kB/s | 46 kB 00:00 2023-12-07T07:23:58.486 INFO:teuthology.orchestra.run.smithi154.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2023-12-07T07:23:58.511 INFO:teuthology.orchestra.run.smithi154.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2023-12-07T07:23:58.528 INFO:teuthology.orchestra.run.smithi154.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2023-12-07T07:23:58.545 INFO:teuthology.orchestra.run.smithi154.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-12-07T07:23:58.561 INFO:teuthology.orchestra.run.smithi154.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2023-12-07T07:23:58.662 INFO:teuthology.orchestra.run.smithi154.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-12-07T07:23:58.687 INFO:teuthology.orchestra.run.smithi154.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2023-12-07T07:23:58.691 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T07:23:58.691 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 1/31 2023-12-07T07:23:58.691 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 2/31 2023-12-07T07:23:58.691 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 3/31 2023-12-07T07:23:58.691 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 4/31 2023-12-07T07:23:58.691 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 5/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 6/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 7/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-64.ge068ebcf.el9.x8 8/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_ 9/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 10/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 11/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 12/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 13/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-07T07:23:58.692 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 28/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-07T07:23:58.693 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 30/31 2023-12-07T07:23:58.704 INFO:teuthology.orchestra.run.smithi154.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 140 kB/s | 20 kB 00:00 2023-12-07T07:23:58.709 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:23:58.709 INFO:teuthology.orchestra.run.smithi154.stdout:Total 12 MB/s | 30 MB 00:02 2023-12-07T07:23:58.721 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:23:58.760 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:23:58.761 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:23:59.178 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:23:59.179 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:23:59.657 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-07T07:23:59.657 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:23:59.657 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2023-12-07T07:23:59.657 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.658 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.658 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:23:59.658 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-07T07:23:59.658 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.658 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.658 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.659 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.659 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-07T07:23:59.659 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-07T07:23:59.659 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-07T07:23:59.659 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.659 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-07T07:23:59.660 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-07T07:23:59.660 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-07T07:23:59.660 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.660 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-07T07:23:59.660 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.660 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-07T07:23:59.660 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-07T07:23:59.661 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-07T07:23:59.661 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-07T07:23:59.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:23:59.662 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2023-12-07T07:23:59.663 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-07T07:23:59.663 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:23:59.663 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:00.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test 2023-12-07T07:24:00.520 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:00.578 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:27 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:00.581 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-07T07:24:00.624 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-07T07:24:00.659 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-07T07:24:00.727 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 43 M 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 164 k 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-07T07:24:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-07T07:24:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:00.729 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:00.729 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:00.729 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2023-12-07T07:24:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:00.729 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-07T07:24:00.730 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 43 M 2023-12-07T07:24:00.730 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 176 M 2023-12-07T07:24:00.730 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:00.774 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-07T07:24:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-07T07:24:00.881 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-07T07:24:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-07T07:24:00.931 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-07T07:24:00.973 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-07T07:24:00.977 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-07T07:24:00.998 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): libcephsqlite-19.0.0-64.ge068ebcf.el9.x8 612 kB/s | 164 kB 00:00 2023-12-07T07:24:01.018 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-07T07:24:01.023 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-12-07T07:24:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-07T07:24:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-07T07:24:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-07T07:24:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-07T07:24:01.299 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-07T07:24:01.481 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-07T07:24:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-07T07:24:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-07T07:24:01.594 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-07T07:24:01.623 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-07T07:24:01.700 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-07T07:24:01.733 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T07:24:01.759 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-07T07:24:01.799 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-07T07:24:01.834 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-07T07:24:01.868 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-07T07:24:01.959 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-07T07:24:02.002 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-07T07:24:02.071 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-07T07:24:02.210 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-07T07:24:02.343 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T07:24:02.503 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-64.ge068ebcf.el9.no 31/36 2023-12-07T07:24:02.529 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 32/36 2023-12-07T07:24:02.554 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 32/36 2023-12-07T07:24:02.554 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:02.554 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-07T07:24:02.555 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T07:24:02.555 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T07:24:02.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:04.122 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 33/36 2023-12-07T07:24:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 33/36 2023-12-07T07:24:04.150 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:04.150 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-07T07:24:04.150 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T07:24:04.150 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T07:24:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:04.323 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): ceph-test-19.0.0-64.ge068ebcf.el9.x86_64 12 MB/s | 43 MB 00:03 2023-12-07T07:24:04.326 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:04.326 INFO:teuthology.orchestra.run.smithi097.stdout:Total 12 MB/s | 43 MB 00:03 2023-12-07T07:24:04.327 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:04.346 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:04.347 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:04.598 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 34/36 2023-12-07T07:24:04.621 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 34/36 2023-12-07T07:24:04.621 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:04.621 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-07T07:24:04.621 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T07:24:04.621 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T07:24:04.621 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:04.704 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:04.704 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:04.815 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 35/36 2023-12-07T07:24:04.837 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 35/36 2023-12-07T07:24:04.838 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:04.838 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-07T07:24:04.838 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T07:24:04.838 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T07:24:04.838 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:04.895 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 36/36 2023-12-07T07:24:05.245 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:05.335 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-07T07:24:05.366 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-07T07:24:05.398 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-07T07:24:05.443 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-07T07:24:05.468 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 5/6 2023-12-07T07:24:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 36/36 2023-12-07T07:24:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 1/36 2023-12-07T07:24:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 2/36 2023-12-07T07:24:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 3/36 2023-12-07T07:24:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 4/36 2023-12-07T07:24:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 5/36 2023-12-07T07:24:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-64.ge068ebcf.el9.no 6/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-07T07:24:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-07T07:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-07T07:24:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-07T07:24:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-07T07:24:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T07:24:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-07T07:24:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-07T07:24:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-07T07:24:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-07T07:24:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-07T07:24:07.563 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-07T07:24:07.564 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-07T07:24:07.565 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-07T07:24:07.566 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-07T07:24:07.567 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-07T07:24:07.567 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:07.567 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:07.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2023-12-07T07:24:08.414 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:51 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:08.509 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:08.561 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:08.561 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:08.561 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:08.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2023-12-07T07:24:09.202 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:52 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:09.347 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:09.347 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:09.347 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:24:09.347 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:09.347 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 802 k 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 802 k 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 817 k 2023-12-07T07:24:09.348 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 5/6 2023-12-07T07:24:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 6/6 2023-12-07T07:24:09.638 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-19.0.0-64.ge068ebcf.el9.noarch.rpm 2.7 MB/s | 802 kB 00:00 2023-12-07T07:24:09.639 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:09.639 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.7 MB/s | 802 kB 00:00 2023-12-07T07:24:09.639 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:09.644 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:09.644 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:09.650 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:09.651 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:09.726 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:10.109 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:10.200 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:10.868 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:10.868 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:11.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2023-12-07T07:24:11.667 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:54 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:11.741 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 6/6 2023-12-07T07:24:11.741 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 1/6 2023-12-07T07:24:11.741 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 2/6 2023-12-07T07:24:11.741 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-07T07:24:11.741 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-07T07:24:11.741 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-07T07:24:11.813 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:11.814 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:11.814 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:24:11.815 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:11.815 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:11.815 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 141 k 2023-12-07T07:24:11.815 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:11.815 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:11.815 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:11.816 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:24:11.816 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:11.816 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 141 k 2023-12-07T07:24:11.816 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 400 k 2023-12-07T07:24:11.816 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:12.051 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-19.0.0-64.ge068ebcf 598 kB/s | 141 kB 00:00 2023-12-07T07:24:12.052 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:12.052 INFO:teuthology.orchestra.run.smithi154.stdout:Total 595 kB/s | 141 kB 00:00 2023-12-07T07:24:12.052 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:12.059 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:12.059 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:12.112 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:12.113 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:12.135 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-07T07:24:12.135 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:12.136 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:12.136 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-64.ge068ebcf.el9.x86_64 jq-1.6-16.el9.x86_64 2023-12-07T07:24:12.136 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-64.ge068ebcf.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-12-07T07:24:12.136 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-12-07T07:24:12.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:12.136 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:12.248 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:12.311 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el 1/1 2023-12-07T07:24:12.335 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el 1/1 2023-12-07T07:24:12.335 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:12.335 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-07T07:24:12.336 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:12.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph 2023-12-07T07:24:13.002 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:39 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:13.154 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 6.5 k 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 2.1 M 2023-12-07T07:24:13.156 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.5 M 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 243 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 4.4 M 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 18 M 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-07T07:24:13.157 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-07T07:24:13.158 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================= 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout:Install 36 Packages 2023-12-07T07:24:13.159 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:13.160 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 30 M 2023-12-07T07:24:13.160 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 111 M 2023-12-07T07:24:13.160 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:13.377 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el 1/1 2023-12-07T07:24:13.377 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:13.377 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:13.377 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:13.378 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:13.378 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:13.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2023-12-07T07:24:13.601 INFO:teuthology.orchestra.run.smithi097.stdout:(1/36): ceph-19.0.0-64.ge068ebcf.el9.x86_64.rpm 39 kB/s | 6.5 kB 00:00 2023-12-07T07:24:13.851 INFO:teuthology.orchestra.run.smithi097.stdout:(2/36): ceph-mds-19.0.0-64.ge068ebcf.el9.x86_64 5.0 MB/s | 2.1 MB 00:00 2023-12-07T07:24:13.918 INFO:teuthology.orchestra.run.smithi097.stdout:(3/36): ceph-mgr-19.0.0-64.ge068ebcf.el9.x86_64 3.0 MB/s | 1.5 MB 00:00 2023-12-07T07:24:14.010 INFO:teuthology.orchestra.run.smithi097.stdout:(4/36): ceph-mgr-modules-core-19.0.0-64.ge068eb 2.6 MB/s | 243 kB 00:00 2023-12-07T07:24:14.175 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:57 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:14.244 INFO:teuthology.orchestra.run.smithi097.stdout:(5/36): ceph-mon-19.0.0-64.ge068ebcf.el9.x86_64 6.8 MB/s | 4.4 MB 00:00 2023-12-07T07:24:14.269 INFO:teuthology.orchestra.run.smithi097.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 722 kB/s | 187 kB 00:00 2023-12-07T07:24:14.271 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:14.322 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:14.323 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:14.323 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:14.353 INFO:teuthology.orchestra.run.smithi097.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2023-12-07T07:24:14.370 INFO:teuthology.orchestra.run.smithi097.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 314 kB/s | 32 kB 00:00 2023-12-07T07:24:14.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2023-12-07T07:24:14.413 INFO:teuthology.orchestra.run.smithi097.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 810 kB/s | 47 kB 00:00 2023-12-07T07:24:14.429 INFO:teuthology.orchestra.run.smithi097.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 659 kB/s | 39 kB 00:00 2023-12-07T07:24:14.471 INFO:teuthology.orchestra.run.smithi097.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-12-07T07:24:14.529 INFO:teuthology.orchestra.run.smithi097.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2023-12-07T07:24:14.555 INFO:teuthology.orchestra.run.smithi097.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-12-07T07:24:14.580 INFO:teuthology.orchestra.run.smithi097.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-07T07:24:14.605 INFO:teuthology.orchestra.run.smithi097.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-07T07:24:14.631 INFO:teuthology.orchestra.run.smithi097.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-07T07:24:14.814 INFO:teuthology.orchestra.run.smithi097.stdout:(17/36): ceph-osd-19.0.0-64.ge068ebcf.el9.x86_6 18 MB/s | 18 MB 00:00 2023-12-07T07:24:14.856 INFO:teuthology.orchestra.run.smithi097.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2023-12-07T07:24:14.957 INFO:teuthology.orchestra.run.smithi097.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 306 kB/s | 43 kB 00:00 2023-12-07T07:24:14.958 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:57 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:15.023 INFO:teuthology.orchestra.run.smithi097.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2023-12-07T07:24:15.057 INFO:teuthology.orchestra.run.smithi097.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 857 kB/s | 172 kB 00:00 2023-12-07T07:24:15.074 INFO:teuthology.orchestra.run.smithi097.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2023-12-07T07:24:15.102 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 1.8 M 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 28 k 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 15 k 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-07T07:24:15.103 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout:Install 7 Packages 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:15.104 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.1 M 2023-12-07T07:24:15.105 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 8.7 M 2023-12-07T07:24:15.105 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:15.108 INFO:teuthology.orchestra.run.smithi097.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2023-12-07T07:24:15.125 INFO:teuthology.orchestra.run.smithi097.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2023-12-07T07:24:15.158 INFO:teuthology.orchestra.run.smithi097.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2023-12-07T07:24:15.175 INFO:teuthology.orchestra.run.smithi097.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2023-12-07T07:24:15.192 INFO:teuthology.orchestra.run.smithi097.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2023-12-07T07:24:15.209 INFO:teuthology.orchestra.run.smithi097.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2023-12-07T07:24:15.259 INFO:teuthology.orchestra.run.smithi097.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2023-12-07T07:24:15.285 INFO:teuthology.orchestra.run.smithi097.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2023-12-07T07:24:15.310 INFO:teuthology.orchestra.run.smithi097.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2023-12-07T07:24:15.335 INFO:teuthology.orchestra.run.smithi097.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2023-12-07T07:24:15.360 INFO:teuthology.orchestra.run.smithi097.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2023-12-07T07:24:15.381 INFO:teuthology.orchestra.run.smithi097.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.2 MB/s | 230 kB 00:00 2023-12-07T07:24:15.406 INFO:teuthology.orchestra.run.smithi097.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 438 kB/s | 20 kB 00:00 2023-12-07T07:24:15.473 INFO:teuthology.orchestra.run.smithi097.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.0 MB/s | 427 kB 00:00 2023-12-07T07:24:15.476 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:15.477 INFO:teuthology.orchestra.run.smithi097.stdout:Total 13 MB/s | 30 MB 00:02 2023-12-07T07:24:15.489 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:15.531 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:15.531 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:15.621 INFO:teuthology.orchestra.run.smithi154.stdout:(1/7): ceph-grafana-dashboards-19.0.0-64.ge068e 156 kB/s | 28 kB 00:00 2023-12-07T07:24:15.672 INFO:teuthology.orchestra.run.smithi154.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 182 kB/s | 9.0 kB 00:00 2023-12-07T07:24:15.705 INFO:teuthology.orchestra.run.smithi154.stdout:(3/7): ceph-prometheus-alerts-19.0.0-64.ge068eb 57 kB/s | 15 kB 00:00 2023-12-07T07:24:15.730 INFO:teuthology.orchestra.run.smithi154.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 698 kB/s | 41 kB 00:00 2023-12-07T07:24:15.789 INFO:teuthology.orchestra.run.smithi154.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 367 kB/s | 31 kB 00:00 2023-12-07T07:24:15.814 INFO:teuthology.orchestra.run.smithi154.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2023-12-07T07:24:15.931 INFO:teuthology.orchestra.run.smithi154.stdout:(7/7): ceph-mgr-dashboard-19.0.0-64.ge068ebcf.e 3.6 MB/s | 1.8 MB 00:00 2023-12-07T07:24:15.932 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:15.933 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.5 MB/s | 2.1 MB 00:00 2023-12-07T07:24:15.933 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:15.933 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:15.934 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:15.940 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:15.940 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:16.008 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:16.008 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:16.308 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-07T07:24:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-07T07:24:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-07T07:24:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-07T07:24:16.508 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-64.ge068ebcf.el9.noa 5/7 2023-12-07T07:24:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-64.ge068ebcf.el9.no 6/7 2023-12-07T07:24:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 7/7 2023-12-07T07:24:17.231 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:17.284 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-07T07:24:17.318 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-07T07:24:17.336 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 7/7 2023-12-07T07:24:17.336 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-64.ge068ebcf.el9.no 1/7 2023-12-07T07:24:17.336 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 2/7 2023-12-07T07:24:17.336 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-64.ge068ebcf.el9.noa 3/7 2023-12-07T07:24:17.336 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-07T07:24:17.336 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-07T07:24:17.337 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-07T07:24:17.345 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-07T07:24:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-07T07:24:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-07T07:24:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-07T07:24:17.586 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:17.646 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:17.672 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-07T07:24:17.713 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-07T07:24:17.785 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-07T07:24:17.838 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-07T07:24:17.877 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-07T07:24:17.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-07T07:24:17.926 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-07T07:24:17.994 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-07T07:24:18.181 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-07T07:24:18.232 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-07T07:24:18.262 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-07T07:24:18.296 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-07T07:24:18.326 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-07T07:24:18.404 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-07T07:24:18.437 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T07:24:18.470 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-07T07:24:18.474 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:01 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:18.511 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-07T07:24:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-07T07:24:18.581 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-07T07:24:18.619 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================== 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================== 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 7.4 M 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-07T07:24:18.620 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================== 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout:Install 12 Packages 2023-12-07T07:24:18.621 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:18.622 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 37 M 2023-12-07T07:24:18.622 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 209 M 2023-12-07T07:24:18.622 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:18.669 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-07T07:24:18.720 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-07T07:24:18.792 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-07T07:24:18.929 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-07T07:24:19.037 INFO:teuthology.orchestra.run.smithi154.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 236 kB/s | 32 kB 00:00 2023-12-07T07:24:19.045 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T07:24:19.071 INFO:teuthology.orchestra.run.smithi154.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2023-12-07T07:24:19.113 INFO:teuthology.orchestra.run.smithi154.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 846 kB/s | 34 kB 00:00 2023-12-07T07:24:19.171 INFO:teuthology.orchestra.run.smithi154.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-12-07T07:24:19.212 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-64.ge068ebcf.el9.no 31/36 2023-12-07T07:24:19.233 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 32/36 2023-12-07T07:24:19.257 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 32/36 2023-12-07T07:24:19.258 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:19.258 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-07T07:24:19.258 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T07:24:19.258 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-07T07:24:19.258 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:19.347 INFO:teuthology.orchestra.run.smithi154.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-12-07T07:24:19.414 INFO:teuthology.orchestra.run.smithi154.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2023-12-07T07:24:19.439 INFO:teuthology.orchestra.run.smithi154.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-07T07:24:19.664 INFO:teuthology.orchestra.run.smithi154.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-64 9.7 MB/s | 7.4 MB 00:00 2023-12-07T07:24:19.823 INFO:teuthology.orchestra.run.smithi154.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2023-12-07T07:24:19.848 INFO:teuthology.orchestra.run.smithi154.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2023-12-07T07:24:19.874 INFO:teuthology.orchestra.run.smithi154.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 943 kB/s | 195 kB 00:00 2023-12-07T07:24:20.007 INFO:teuthology.orchestra.run.smithi154.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2023-12-07T07:24:20.009 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:20.010 INFO:teuthology.orchestra.run.smithi154.stdout:Total 27 MB/s | 37 MB 00:01 2023-12-07T07:24:20.023 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:20.047 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:20.047 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:20.451 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:20.451 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:20.785 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 33/36 2023-12-07T07:24:20.813 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 33/36 2023-12-07T07:24:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-07T07:24:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T07:24:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-07T07:24:20.813 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:21.148 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:21.222 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-07T07:24:21.253 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-07T07:24:21.286 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 34/36 2023-12-07T07:24:21.310 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 34/36 2023-12-07T07:24:21.310 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:21.310 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-07T07:24:21.310 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T07:24:21.310 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-07T07:24:21.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:21.347 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-07T07:24:21.381 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-07T07:24:21.521 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 35/36 2023-12-07T07:24:21.544 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 35/36 2023-12-07T07:24:21.544 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:21.544 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-07T07:24:21.544 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T07:24:21.544 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-07T07:24:21.544 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:21.569 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T07:24:21.597 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T07:24:21.603 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 36/36 2023-12-07T07:24:21.674 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-07T07:24:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-07T07:24:22.443 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-07T07:24:23.471 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 36/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 1/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 2/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 3/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 4/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 5/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-64.ge068ebcf.el9.no 6/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-07T07:24:23.472 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-07T07:24:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-07T07:24:23.474 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-07T07:24:23.708 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-07T07:24:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2023-12-07T07:24:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-07T07:24:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-07T07:24:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-07T07:24:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:24.372 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:24.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base 2023-12-07T07:24:24.737 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-07T07:24:24.763 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebc 12/12 2023-12-07T07:24:25.316 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:52 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:25.410 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:25.459 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:25.460 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:24:25.460 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:25.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install cephadm 2023-12-07T07:24:26.087 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:53 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:26.230 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:26.231 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:26.231 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:26.231 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:26.231 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:26.231 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 802 k 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 802 k 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 817 k 2023-12-07T07:24:26.232 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:26.764 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm-19.0.0-64.ge068ebcf.el9.noarch.rpm 1.5 MB/s | 802 kB 00:00 2023-12-07T07:24:26.764 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:26.764 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.5 MB/s | 802 kB 00:00 2023-12-07T07:24:26.764 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:26.769 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:26.770 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:26.776 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:26.776 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:26.858 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:27.244 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:27.268 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebc 12/12 2023-12-07T07:24:27.268 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebc 1/12 2023-12-07T07:24:27.268 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-07T07:24:27.268 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-07T07:24:27.268 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-07T07:24:27.269 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T07:24:27.269 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T07:24:27.269 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-07T07:24:27.269 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-07T07:24:27.269 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-07T07:24:27.269 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-07T07:24:27.269 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-07T07:24:27.336 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:27.697 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-07T07:24:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-07T07:24:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-07T07:24:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-07T07:24:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-07T07:24:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:27.704 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:28.020 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:28.020 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:28.020 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:28.021 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:28.021 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:28.021 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:28.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2023-12-07T07:24:28.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache 2023-12-07T07:24:28.622 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:11 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:28.769 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 50 k 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-07T07:24:28.771 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2023-12-07T07:24:28.772 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:28.773 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.0 M 2023-12-07T07:24:28.773 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 26 M 2023-12-07T07:24:28.773 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:28.810 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:55 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:28.953 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 141 k 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:24:28.954 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 141 k 2023-12-07T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 400 k 2023-12-07T07:24:28.955 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:29.086 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 187 kB/s | 24 kB 00:00 2023-12-07T07:24:29.111 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2023-12-07T07:24:29.162 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-12-07T07:24:29.178 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-19.0.0-64.ge068ebcf 634 kB/s | 141 kB 00:00 2023-12-07T07:24:29.178 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:29.179 INFO:teuthology.orchestra.run.smithi097.stdout:Total 629 kB/s | 141 kB 00:00 2023-12-07T07:24:29.179 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:29.186 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:29.186 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:29.187 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-07T07:24:29.212 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-07T07:24:29.237 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:29.238 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-12-07T07:24:29.238 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:29.263 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): ceph-mgr-rook-19.0.0-64.ge068ebcf.el9.n 164 kB/s | 50 kB 00:00 2023-12-07T07:24:29.370 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:29.437 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el 1/1 2023-12-07T07:24:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el 1/1 2023-12-07T07:24:29.461 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:29.461 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-07T07:24:29.461 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:29.538 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2023-12-07T07:24:29.563 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2023-12-07T07:24:29.772 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2023-12-07T07:24:30.014 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 263 kB/s | 197 kB 00:00 2023-12-07T07:24:30.064 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 307 kB/s | 90 kB 00:00 2023-12-07T07:24:30.473 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el 1/1 2023-12-07T07:24:30.473 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:30.473 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:30.473 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:30.473 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:30.473 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:30.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr 2023-12-07T07:24:31.263 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:58 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:31.358 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-mgr-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:31.407 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:31.408 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:24:31.408 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:31.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-dashboard 2023-12-07T07:24:32.050 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:59 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:32.131 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 403 kB/s | 1.0 MB 00:02 2023-12-07T07:24:32.133 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:32.133 INFO:teuthology.orchestra.run.smithi154.stdout:Total 622 kB/s | 2.0 MB 00:03 2023-12-07T07:24:32.134 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:32.148 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:32.148 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:32.194 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 1.8 M 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 28 k 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 15 k 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-07T07:24:32.195 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout:Install 7 Packages 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 2.1 M 2023-12-07T07:24:32.196 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 8.7 M 2023-12-07T07:24:32.197 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:32.255 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:32.256 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:32.490 INFO:teuthology.orchestra.run.smithi097.stdout:(1/7): ceph-grafana-dashboards-19.0.0-64.ge068e 115 kB/s | 28 kB 00:00 2023-12-07T07:24:32.524 INFO:teuthology.orchestra.run.smithi097.stdout:(2/7): ceph-prometheus-alerts-19.0.0-64.ge068eb 55 kB/s | 15 kB 00:00 2023-12-07T07:24:32.557 INFO:teuthology.orchestra.run.smithi097.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2023-12-07T07:24:32.608 INFO:teuthology.orchestra.run.smithi097.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 613 kB/s | 31 kB 00:00 2023-12-07T07:24:32.624 INFO:teuthology.orchestra.run.smithi097.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 407 kB/s | 41 kB 00:00 2023-12-07T07:24:32.691 INFO:teuthology.orchestra.run.smithi097.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2023-12-07T07:24:32.773 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:32.808 INFO:teuthology.orchestra.run.smithi097.stdout:(7/7): ceph-mgr-dashboard-19.0.0-64.ge068ebcf.e 3.1 MB/s | 1.8 MB 00:00 2023-12-07T07:24:32.810 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:32.810 INFO:teuthology.orchestra.run.smithi097.stdout:Total 3.4 MB/s | 2.1 MB 00:00 2023-12-07T07:24:32.810 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:32.817 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:32.817 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:32.880 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:32.880 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:32.952 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-07T07:24:33.006 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-07T07:24:33.064 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-07T07:24:33.101 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-07T07:24:33.142 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-07T07:24:33.175 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-07T07:24:33.253 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-07T07:24:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-07T07:24:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-07T07:24:33.347 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-07T07:24:33.353 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-07T07:24:33.377 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-64.ge068ebcf.el9.noa 5/7 2023-12-07T07:24:33.402 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-07T07:24:33.615 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-64.ge068ebcf.el9.no 6/7 2023-12-07T07:24:33.645 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 7/7 2023-12-07T07:24:34.130 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-07T07:24:34.174 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-07T07:24:34.202 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 7/7 2023-12-07T07:24:34.202 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-64.ge068ebcf.el9.no 1/7 2023-12-07T07:24:34.202 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 2/7 2023-12-07T07:24:34.202 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-64.ge068ebcf.el9.noa 3/7 2023-12-07T07:24:34.203 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-07T07:24:34.203 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-07T07:24:34.203 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-07T07:24:34.207 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-07T07:24:34.255 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-07T07:24:34.278 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 13/13 2023-12-07T07:24:34.525 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:34.526 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:34.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-07T07:24:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 13/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 1/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-07T07:24:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-07T07:24:35.329 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:02 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:35.440 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-07T07:24:35.440 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:35.440 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:35.440 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:35.440 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-07T07:24:35.440 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-07T07:24:35.440 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:35.441 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:35.473 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:35.474 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================== 2023-12-07T07:24:35.474 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:35.474 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================== 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 7.4 M 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-07T07:24:35.475 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:35.476 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:35.476 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================== 2023-12-07T07:24:35.476 INFO:teuthology.orchestra.run.smithi097.stdout:Install 12 Packages 2023-12-07T07:24:35.476 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:35.476 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 37 M 2023-12-07T07:24:35.476 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 209 M 2023-12-07T07:24:35.476 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:35.649 INFO:teuthology.orchestra.run.smithi097.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 242 kB/s | 32 kB 00:00 2023-12-07T07:24:35.682 INFO:teuthology.orchestra.run.smithi097.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2023-12-07T07:24:35.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2023-12-07T07:24:35.724 INFO:teuthology.orchestra.run.smithi097.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 34 kB 00:00 2023-12-07T07:24:35.783 INFO:teuthology.orchestra.run.smithi097.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-12-07T07:24:35.983 INFO:teuthology.orchestra.run.smithi097.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-12-07T07:24:36.050 INFO:teuthology.orchestra.run.smithi097.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2023-12-07T07:24:36.075 INFO:teuthology.orchestra.run.smithi097.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2023-12-07T07:24:36.242 INFO:teuthology.orchestra.run.smithi097.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2023-12-07T07:24:36.320 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:19 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:36.418 INFO:teuthology.orchestra.run.smithi097.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2023-12-07T07:24:36.443 INFO:teuthology.orchestra.run.smithi097.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 985 kB/s | 195 kB 00:00 2023-12-07T07:24:36.469 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 133 k 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-07T07:24:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-07T07:24:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-07T07:24:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-07T07:24:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:36.471 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:36.471 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:24:36.471 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2023-12-07T07:24:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:36.472 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 6.8 M 2023-12-07T07:24:36.472 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 33 M 2023-12-07T07:24:36.472 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:36.601 INFO:teuthology.orchestra.run.smithi097.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2023-12-07T07:24:36.692 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-07T07:24:36.900 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2023-12-07T07:24:37.059 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2023-12-07T07:24:37.084 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 315 kB/s | 58 kB 00:00 2023-12-07T07:24:37.109 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2023-12-07T07:24:37.135 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): ceph-mgr-cephadm-19.0.0-64.ge068ebcf.el9 217 kB/s | 133 kB 00:00 2023-12-07T07:24:37.138 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:37.138 INFO:teuthology.orchestra.run.smithi154.stdout:Total 10 MB/s | 6.8 MB 00:00 2023-12-07T07:24:37.138 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:37.147 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:37.147 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:37.235 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:37.235 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:37.690 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-07T07:24:37.744 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-07T07:24:37.776 INFO:teuthology.orchestra.run.smithi097.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-6 3.3 MB/s | 7.4 MB 00:02 2023-12-07T07:24:37.779 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:37.779 INFO:teuthology.orchestra.run.smithi097.stdout:Total 16 MB/s | 37 MB 00:02 2023-12-07T07:24:37.793 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:37.817 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:37.817 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:38.205 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:38.205 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:38.256 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-07T07:24:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-07T07:24:38.407 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-07T07:24:38.431 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 6/6 2023-12-07T07:24:38.892 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:38.960 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-07T07:24:38.989 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-07T07:24:39.091 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-07T07:24:39.134 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-07T07:24:39.237 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 6/6 2023-12-07T07:24:39.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/6 2023-12-07T07:24:39.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-07T07:24:39.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-07T07:24:39.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-07T07:24:39.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-07T07:24:39.317 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T07:24:39.350 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T07:24:39.427 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-07T07:24:39.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:39.556 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:39.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2023-12-07T07:24:40.071 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-07T07:24:40.187 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-07T07:24:40.429 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:23 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:40.574 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 864 k 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2023-12-07T07:24:40.575 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:40.576 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 942 k 2023-12-07T07:24:40.576 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.7 M 2023-12-07T07:24:40.576 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:40.702 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 624 kB/s | 79 kB 00:00 2023-12-07T07:24:41.111 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): ceph-fuse-19.0.0-64.ge068ebcf.el9.x86_64 1.6 MB/s | 864 kB 00:00 2023-12-07T07:24:41.112 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:41.113 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.7 MB/s | 942 kB 00:00 2023-12-07T07:24:41.113 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:41.134 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:41.134 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:41.214 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:41.214 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:41.400 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:41.424 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-07T07:24:41.558 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-07T07:24:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:24:42.278 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:24:42.278 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/2 2023-12-07T07:24:42.455 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-07T07:24:42.483 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebc 12/12 2023-12-07T07:24:42.588 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-07T07:24:42.588 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:42.588 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:42.588 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-07T07:24:42.588 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:42.588 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:42.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-volume 2023-12-07T07:24:43.478 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:26 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:43.624 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:43.624 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:43.624 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 262 k 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:43.625 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 262 k 2023-12-07T07:24:43.626 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 1.2 M 2023-12-07T07:24:43.626 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:43.873 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-volume-19.0.0-64.ge068ebcf.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2023-12-07T07:24:43.873 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:43.873 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-12-07T07:24:43.874 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:43.881 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:43.881 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:43.910 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:43.910 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:44.120 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:44.183 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:44.198 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:44.198 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:24:44.199 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-07T07:24:44.199 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:44.917 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebc 12/12 2023-12-07T07:24:44.917 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebc 1/12 2023-12-07T07:24:44.917 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-07T07:24:44.917 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-07T07:24:44.917 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-07T07:24:44.918 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-07T07:24:44.918 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-07T07:24:44.918 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-07T07:24:44.918 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-07T07:24:44.918 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-07T07:24:44.918 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-07T07:24:44.918 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-07T07:24:45.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:24:45.262 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:45.262 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:45.262 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:45.262 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:45.262 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-07T07:24:45.393 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-07T07:24:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-07T07:24:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-07T07:24:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-07T07:24:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-07T07:24:45.394 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:45.394 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:45.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2023-12-07T07:24:45.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-rook 2023-12-07T07:24:46.189 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:29 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:46.336 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:46.336 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:46.336 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:24:46.336 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 127 k 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 127 k 2023-12-07T07:24:46.337 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 456 k 2023-12-07T07:24:46.338 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:46.386 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:13 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:46.532 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:46.533 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:24:46.533 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:46.533 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 50 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-07T07:24:46.534 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-07T07:24:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-07T07:24:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:46.535 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:46.535 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:24:46.535 INFO:teuthology.orchestra.run.smithi097.stdout:Install 13 Packages 2023-12-07T07:24:46.535 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:46.535 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 2.0 M 2023-12-07T07:24:46.536 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 26 M 2023-12-07T07:24:46.536 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:46.569 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-19.0.0-64.ge068ebcf.el9.x86_64.r 547 kB/s | 127 kB 00:00 2023-12-07T07:24:46.570 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:46.570 INFO:teuthology.orchestra.run.smithi154.stdout:Total 543 kB/s | 127 kB 00:00 2023-12-07T07:24:46.570 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:46.576 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:46.576 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:46.630 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:46.631 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:46.827 INFO:teuthology.orchestra.run.smithi097.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 141 kB/s | 18 kB 00:00 2023-12-07T07:24:46.852 INFO:teuthology.orchestra.run.smithi097.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 159 kB/s | 24 kB 00:00 2023-12-07T07:24:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:46.902 INFO:teuthology.orchestra.run.smithi097.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-12-07T07:24:46.928 INFO:teuthology.orchestra.run.smithi097.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-07T07:24:46.953 INFO:teuthology.orchestra.run.smithi097.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-07T07:24:46.978 INFO:teuthology.orchestra.run.smithi097.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2023-12-07T07:24:47.004 INFO:teuthology.orchestra.run.smithi097.stdout:(7/13): ceph-mgr-rook-19.0.0-64.ge068ebcf.el9.n 164 kB/s | 50 kB 00:00 2023-12-07T07:24:47.229 INFO:teuthology.orchestra.run.smithi097.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2023-12-07T07:24:47.254 INFO:teuthology.orchestra.run.smithi097.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2023-12-07T07:24:47.336 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:47.479 INFO:teuthology.orchestra.run.smithi097.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 263 kB/s | 59 kB 00:00 2023-12-07T07:24:47.496 INFO:teuthology.orchestra.run.smithi097.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 401 kB/s | 197 kB 00:00 2023-12-07T07:24:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:47.646 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:47.646 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:47.655 INFO:teuthology.orchestra.run.smithi097.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 511 kB/s | 90 kB 00:00 2023-12-07T07:24:47.755 INFO:teuthology.orchestra.run.smithi097.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2023-12-07T07:24:47.757 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:47.757 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2023-12-07T07:24:47.758 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:47.772 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:47.772 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:47.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2023-12-07T07:24:47.875 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:47.877 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:48.392 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:48.477 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:31 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:48.586 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-07T07:24:48.625 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:48.626 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:48.626 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:48.648 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-07T07:24:48.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2023-12-07T07:24:48.703 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-07T07:24:48.733 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-07T07:24:48.774 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-07T07:24:48.867 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-07T07:24:48.977 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-07T07:24:49.034 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-07T07:24:49.292 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:32 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 31 k 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:49.437 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:49.438 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:24:49.438 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:49.438 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 31 k 2023-12-07T07:24:49.438 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 140 k 2023-12-07T07:24:49.439 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:49.705 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-19.0.0-64.ge068ebcf.el9.x86_64. 117 kB/s | 31 kB 00:00 2023-12-07T07:24:49.706 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:49.706 INFO:teuthology.orchestra.run.smithi154.stdout:Total 116 kB/s | 31 kB 00:00 2023-12-07T07:24:49.706 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:49.710 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:49.710 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:49.734 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:49.734 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:49.754 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-07T07:24:49.806 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-07T07:24:49.824 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:49.840 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-07T07:24:49.888 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-07T07:24:49.910 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 13/13 2023-12-07T07:24:49.920 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:50.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:50.578 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:50.578 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:50.578 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:50.578 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:50.578 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:50.729 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 13/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 1/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-07T07:24:50.730 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-07T07:24:50.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2023-12-07T07:24:51.117 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-07T07:24:51.117 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:51.117 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:51.117 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:51.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-07T07:24:51.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-07T07:24:51.117 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:51.118 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:51.368 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:34 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:51.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-cephadm 2023-12-07T07:24:51.464 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:51.513 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:51.514 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:51.514 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:51.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2023-12-07T07:24:51.970 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:18 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:52.115 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 133 k 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-07T07:24:52.116 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:52.117 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:52.117 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:24:52.117 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2023-12-07T07:24:52.117 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:52.117 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 6.8 M 2023-12-07T07:24:52.117 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 33 M 2023-12-07T07:24:52.117 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:52.168 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:35 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:52.264 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:52.314 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:52.315 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:52.315 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:52.351 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-07T07:24:52.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2023-12-07T07:24:52.509 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-12-07T07:24:52.718 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): ceph-mgr-cephadm-19.0.0-64.ge068ebcf.el9 255 kB/s | 133 kB 00:00 2023-12-07T07:24:52.785 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 211 kB/s | 58 kB 00:00 2023-12-07T07:24:52.835 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2023-12-07T07:24:52.972 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:35 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:53.010 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2023-12-07T07:24:53.014 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:53.015 INFO:teuthology.orchestra.run.smithi097.stdout:Total 7.5 MB/s | 6.8 MB 00:00 2023-12-07T07:24:53.015 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:53.023 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:53.023 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:53.066 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:53.111 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:53.111 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:53.118 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:53.119 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:53.119 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:53.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2023-12-07T07:24:53.420 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:53.581 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-07T07:24:53.628 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-07T07:24:53.777 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:36 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:53.873 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:53.925 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:53.926 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:53.926 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:53.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2023-12-07T07:24:54.135 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-07T07:24:54.230 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-07T07:24:54.292 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-07T07:24:54.316 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 6/6 2023-12-07T07:24:54.580 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:37 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:54.674 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:54.724 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:54.725 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:54.725 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:54.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2023-12-07T07:24:55.147 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 6/6 2023-12-07T07:24:55.147 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 1/6 2023-12-07T07:24:55.147 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-07T07:24:55.147 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-07T07:24:55.148 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-07T07:24:55.148 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-07T07:24:55.376 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:38 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:55.472 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:24:55.475 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-07T07:24:55.475 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:55.475 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:55.476 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:24:55.476 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-07T07:24:55.476 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-07T07:24:55.476 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-07T07:24:55.476 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-07T07:24:55.476 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-07T07:24:55.477 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:55.477 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:55.524 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:55.525 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:24:55.526 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:55.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2023-12-07T07:24:55.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse 2023-12-07T07:24:56.189 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:39 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:56.293 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:23 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:56.337 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 85 k 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 85 k 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 231 k 2023-12-07T07:24:56.338 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:56.439 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:56.439 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 864 k 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2023-12-07T07:24:56.440 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:56.441 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 942 k 2023-12-07T07:24:56.441 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.7 M 2023-12-07T07:24:56.441 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:24:56.575 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-19.0.0-64.ge068ebcf.el9.x86_64.rpm 361 kB/s | 85 kB 00:00 2023-12-07T07:24:56.576 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:56.576 INFO:teuthology.orchestra.run.smithi154.stdout:Total 359 kB/s | 85 kB 00:00 2023-12-07T07:24:56.576 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:24:56.579 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 569 kB/s | 79 kB 00:00 2023-12-07T07:24:56.583 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:24:56.583 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:24:56.639 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:24:56.639 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:24:56.775 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:24:56.872 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:57.163 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-fuse-19.0.0-64.ge068ebcf.el9.x86_64 1.2 MB/s | 864 kB 00:00 2023-12-07T07:24:57.164 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:24:57.164 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.3 MB/s | 942 kB 00:00 2023-12-07T07:24:57.164 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:24:57.185 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:24:57.185 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:24:57.258 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:57.265 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:24:57.265 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:24:57.452 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:24:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:24:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:57.555 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:24:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:24:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:57.555 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:24:57.605 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-07T07:24:57.671 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:24:57.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2023-12-07T07:24:58.386 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:41 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:24:58.405 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:24:58.406 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/2 2023-12-07T07:24:58.532 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:24:58.532 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 3.0 M 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.0 M 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 12 M 2023-12-07T07:24:58.533 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:24:58.729 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-07T07:24:58.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:58.729 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:24:58.729 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-07T07:24:58.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:58.729 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:24:59.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-volume 2023-12-07T07:24:59.610 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:26 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:24:59.756 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:24:59.758 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:59.758 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2023-12-07T07:24:59.758 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:59.758 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:24:59.758 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-64.ge068ebcf.el9 ceph-noarch 262 k 2023-12-07T07:24:59.759 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:59.759 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:24:59.759 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:24:59.759 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:24:59.759 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:24:59.759 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 262 k 2023-12-07T07:24:59.759 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.2 M 2023-12-07T07:24:59.760 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:00.027 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-19.0.0-64.ge068ebcf.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2023-12-07T07:25:00.028 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:00.028 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.0 MB/s | 3.0 MB 00:01 2023-12-07T07:25:00.029 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:25:00.039 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:25:00.039 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:25:00.114 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:25:00.115 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:25:00.168 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-volume-19.0.0-64.ge068ebcf.el9.noarch.rpm 641 kB/s | 262 kB 00:00 2023-12-07T07:25:00.168 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:00.168 INFO:teuthology.orchestra.run.smithi097.stdout:Total 638 kB/s | 262 kB 00:00 2023-12-07T07:25:00.169 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:25:00.177 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:25:00.177 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:25:00.206 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:25:00.206 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:25:00.421 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:25:00.486 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:25:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:25:00.502 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:25:00.502 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-07T07:25:00.502 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:00.535 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:25:00.595 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:00.619 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:00.619 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:25:00.619 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-07T07:25:00.620 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T07:25:00.620 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T07:25:00.620 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:01.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 1/1 2023-12-07T07:25:01.556 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:01.556 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:25:01.556 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-64.ge068ebcf.el9.noarch 2023-12-07T07:25:01.556 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:01.556 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:01.759 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:01.759 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:01.759 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:25:01.759 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:01.759 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:01.759 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:25:01.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel 2023-12-07T07:25:02.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2023-12-07T07:25:02.440 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:29 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:02.586 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 127 k 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 127 k 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 456 k 2023-12-07T07:25:02.587 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:02.621 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:45 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:25:02.770 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 170 k 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 170 k 2023-12-07T07:25:02.771 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 494 k 2023-12-07T07:25:02.772 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:25:02.816 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-19.0.0-64.ge068ebcf.el9.x86_64.r 556 kB/s | 127 kB 00:00 2023-12-07T07:25:02.816 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:02.816 INFO:teuthology.orchestra.run.smithi097.stdout:Total 552 kB/s | 127 kB 00:00 2023-12-07T07:25:02.817 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:25:02.822 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:25:02.822 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:25:02.876 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:25:02.877 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:25:03.016 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:25:03.122 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-19.0.0-64.ge068ebcf.el9.x86_64.rpm 487 kB/s | 170 kB 00:00 2023-12-07T07:25:03.122 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:03.122 INFO:teuthology.orchestra.run.smithi154.stdout:Total 485 kB/s | 170 kB 00:00 2023-12-07T07:25:03.122 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:25:03.130 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:03.130 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:25:03.130 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:25:03.186 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:25:03.186 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:25:03.334 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:25:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:03.548 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:03.838 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:03.838 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:03.838 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:25:03.838 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:03.838 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:03.838 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:03.870 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:04.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2 2023-12-07T07:25:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:04.166 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:25:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:04.166 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:25:04.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base-debuginfo 2023-12-07T07:25:04.670 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:31 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:04.770 INFO:teuthology.orchestra.run.smithi097.stdout:Package libcephfs2-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:25:04.821 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:04.822 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:25:04.822 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:04.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs-devel 2023-12-07T07:25:04.992 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:47 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:25:05.150 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:25:05.150 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 73 M 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 21 M 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2023-12-07T07:25:05.151 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:05.152 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 94 M 2023-12-07T07:25:05.152 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 514 M 2023-12-07T07:25:05.152 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:25:05.464 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:32 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:05.608 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:05.608 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 31 k 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 31 k 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 140 k 2023-12-07T07:25:05.609 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:05.870 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs-devel-19.0.0-64.ge068ebcf.el9.x86_64. 119 kB/s | 31 kB 00:00 2023-12-07T07:25:05.870 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:05.871 INFO:teuthology.orchestra.run.smithi097.stdout:Total 119 kB/s | 31 kB 00:00 2023-12-07T07:25:05.871 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:25:05.875 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:25:05.875 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:25:05.896 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:25:05.897 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:25:05.992 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:25:06.128 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:06.515 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:06.778 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:06.779 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:06.779 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:25:06.779 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:06.779 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:06.779 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:06.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2 2023-12-07T07:25:07.544 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:34 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:07.640 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:25:07.690 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:07.691 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:25:07.691 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:07.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1 2023-12-07T07:25:08.337 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:35 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:08.433 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:25:08.483 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:25:08.484 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:08.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados 2023-12-07T07:25:08.725 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): ceph-debuginfo-19.0.0-64.ge068ebcf.el9.x 5.9 MB/s | 21 MB 00:03 2023-12-07T07:25:09.129 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:36 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:09.225 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rados-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:25:09.276 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:09.277 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:25:09.277 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:09.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw 2023-12-07T07:25:09.408 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): ceph-base-debuginfo-19.0.0-64.ge068ebcf. 17 MB/s | 73 MB 00:04 2023-12-07T07:25:09.410 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:09.411 INFO:teuthology.orchestra.run.smithi154.stdout:Total 22 MB/s | 94 MB 00:04 2023-12-07T07:25:09.411 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:25:09.416 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:25:09.416 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:25:09.920 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:36 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:10.015 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rgw-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:25:10.026 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:25:10.026 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:25:10.066 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:10.067 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:25:10.067 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:10.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs 2023-12-07T07:25:10.714 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:37 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:10.810 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-cephfs-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:25:10.860 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:10.861 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:25:10.861 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:10.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd 2023-12-07T07:25:11.506 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:38 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:11.600 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rbd-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:25:11.652 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:11.652 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:25:11.652 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:11.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse 2023-12-07T07:25:12.307 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:39 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:12.452 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 85 k 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:12.453 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 85 k 2023-12-07T07:25:12.454 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 231 k 2023-12-07T07:25:12.454 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:12.773 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-19.0.0-64.ge068ebcf.el9.x86_64.rpm 267 kB/s | 85 kB 00:00 2023-12-07T07:25:12.774 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:12.774 INFO:teuthology.orchestra.run.smithi097.stdout:Total 266 kB/s | 85 kB 00:00 2023-12-07T07:25:12.774 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:25:12.781 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:25:12.781 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:25:12.835 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:25:12.835 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:25:12.975 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:25:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:25:13.106 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:13.488 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:13.794 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:13.794 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:13.794 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:25:13.795 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:13.795 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:13.795 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:14.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror 2023-12-07T07:25:14.612 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:41 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:14.760 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:14.760 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:14.760 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2023-12-07T07:25:14.760 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 3.0 M 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.0 M 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 12 M 2023-12-07T07:25:14.761 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:15.160 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-19.0.0-64.ge068ebcf.el9.x86_64.rpm 7.6 MB/s | 3.0 MB 00:00 2023-12-07T07:25:15.160 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:15.161 INFO:teuthology.orchestra.run.smithi097.stdout:Total 7.6 MB/s | 3.0 MB 00:00 2023-12-07T07:25:15.161 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:25:15.170 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:25:15.170 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:25:15.242 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:25:15.242 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:25:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:25:15.702 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:15.727 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-07T07:25:15.727 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-07T07:25:15.727 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T07:25:15.727 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-07T07:25:15.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:16.852 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:16.852 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:16.852 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:25:16.853 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:16.853 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:16.853 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:17.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd 2023-12-07T07:25:17.704 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:44 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:17.850 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:17.851 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:17.851 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 170 k 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:25:17.852 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:17.853 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 170 k 2023-12-07T07:25:17.853 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 494 k 2023-12-07T07:25:17.853 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:18.073 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-19.0.0-64.ge068ebcf.el9.x86_64.rpm 772 kB/s | 170 kB 00:00 2023-12-07T07:25:18.073 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:18.073 INFO:teuthology.orchestra.run.smithi097.stdout:Total 768 kB/s | 170 kB 00:00 2023-12-07T07:25:18.074 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:25:18.082 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:25:18.082 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:25:18.136 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:25:18.137 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:25:18.285 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:25:18.396 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:18.820 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:19.143 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:25:19.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:19.143 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:25:19.144 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:19.144 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:19.144 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:19.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base-debuginfo 2023-12-07T07:25:19.955 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:46 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:20.104 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 73 M 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 21 M 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2023-12-07T07:25:20.105 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:20.106 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 94 M 2023-12-07T07:25:20.106 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 514 M 2023-12-07T07:25:20.106 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:20.660 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/2 2023-12-07T07:25:20.732 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:25:21.568 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): ceph-debuginfo-19.0.0-64.ge068ebcf.el9.x 14 MB/s | 21 MB 00:01 2023-12-07T07:25:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:25:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/2 2023-12-07T07:25:24.401 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-base-debuginfo-19.0.0-64.ge068ebcf. 17 MB/s | 73 MB 00:04 2023-12-07T07:25:24.404 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:24.404 INFO:teuthology.orchestra.run.smithi097.stdout:Total 22 MB/s | 94 MB 00:04 2023-12-07T07:25:24.404 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:25:24.409 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:25:24.409 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:25:24.467 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:25:24.468 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:24.468 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:25:24.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:24.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:24.468 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:24.468 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:25:24.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-common-debuginfo 2023-12-07T07:25:25.025 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:25:25.025 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:25:25.256 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:08 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:25:25.403 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 417 M 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:25:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:25:25.405 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 417 M 2023-12-07T07:25:25.405 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 1.7 G 2023-12-07T07:25:25.405 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:25:28.014 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:25:35.699 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/2 2023-12-07T07:25:35.860 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:25:39.331 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:25:39.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/2 2023-12-07T07:25:39.616 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2/2 2023-12-07T07:25:39.616 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:39.616 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:25:39.616 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:39.617 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:25:39.617 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:39.617 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:25:39.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-common-debuginfo 2023-12-07T07:25:40.405 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:07 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:25:40.550 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 417 M 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:25:40.551 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:25:40.552 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:25:40.552 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:25:40.552 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 417 M 2023-12-07T07:25:40.552 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.7 G 2023-12-07T07:25:40.552 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:25:43.602 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-common-debuginfo-19.0.0-64.ge068ebcf.el9.x 23 MB/s | 417 MB 00:18 2023-12-07T07:25:43.603 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:25:43.603 INFO:teuthology.orchestra.run.smithi154.stdout:Total 23 MB/s | 417 MB 00:18 2023-12-07T07:25:43.604 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:25:43.608 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:25:43.608 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:25:46.080 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:25:46.080 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:26:01.888 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-common-debuginfo-19.0.0-64.ge068ebcf.el9.x 20 MB/s | 417 MB 00:21 2023-12-07T07:26:01.889 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:26:01.889 INFO:teuthology.orchestra.run.smithi097.stdout:Total 20 MB/s | 417 MB 00:21 2023-12-07T07:26:01.890 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:26:01.894 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:26:01.894 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:26:04.381 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:26:04.381 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:26:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:26:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:26:36.415 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:26:36.695 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:26:36.773 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:36.773 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:26:36.773 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:26:36.773 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:36.773 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:26:36.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-07T07:26:37.536 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:03:20 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.4 M 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2023-12-07T07:26:37.682 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:26:37.683 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:37.683 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 1.4 M 2023-12-07T07:26:37.683 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 6.2 M 2023-12-07T07:26:37.683 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:26:38.158 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-64 2.9 MB/s | 1.4 MB 00:00 2023-12-07T07:26:38.158 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:26:38.159 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.9 MB/s | 1.4 MB 00:00 2023-12-07T07:26:38.159 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:26:38.162 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:26:38.163 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:26:38.205 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:26:38.206 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:26:38.464 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:26:38.558 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge 1/1 2023-12-07T07:26:38.976 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge 1/1 2023-12-07T07:26:39.234 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge 1/1 2023-12-07T07:26:39.234 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:39.234 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:26:39.234 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:26:39.234 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:39.234 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:26:39.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-07T07:26:40.004 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:03:23 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:26:40.149 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 178 M 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:26:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 178 M 2023-12-07T07:26:40.151 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 768 M 2023-12-07T07:26:40.151 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:26:46.530 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:26:46.650 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:26:48.135 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-radosgw-debuginfo-19.0.0-64.ge068ebcf.el9. 22 MB/s | 178 MB 00:07 2023-12-07T07:26:48.136 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:26:48.136 INFO:teuthology.orchestra.run.smithi154.stdout:Total 22 MB/s | 178 MB 00:07 2023-12-07T07:26:48.137 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:26:48.141 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:26:48.141 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:26:49.224 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:26:49.224 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:26:55.644 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:26:55.921 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:26:55.921 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:55.921 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:26:55.921 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:26:55.921 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:55.921 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:26:56.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-07T07:26:56.751 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:23 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:26:56.906 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:26:56.907 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:26:56.907 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:26:56.907 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:26:56.907 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:26:56.907 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.4 M 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.4 M 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 6.2 M 2023-12-07T07:26:56.908 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:26:57.222 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-64 4.4 MB/s | 1.4 MB 00:00 2023-12-07T07:26:57.223 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:26:57.224 INFO:teuthology.orchestra.run.smithi097.stdout:Total 4.4 MB/s | 1.4 MB 00:00 2023-12-07T07:26:57.224 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:26:57.227 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:26:57.227 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:26:57.274 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:26:57.274 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:26:57.529 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:26:57.625 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge 1/1 2023-12-07T07:26:58.031 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge 1/1 2023-12-07T07:26:58.296 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge 1/1 2023-12-07T07:26:58.296 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:58.296 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:26:58.296 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:26:58.296 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:58.296 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:26:58.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-07T07:26:59.050 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:26 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:26:59.194 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 178 M 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:26:59.195 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:26:59.196 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 178 M 2023-12-07T07:26:59.196 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 768 M 2023-12-07T07:26:59.196 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:27:07.299 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-radosgw-debuginfo-19.0.0-64.ge068ebcf.el9. 22 MB/s | 178 MB 00:08 2023-12-07T07:27:07.300 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:27:07.301 INFO:teuthology.orchestra.run.smithi097.stdout:Total 22 MB/s | 178 MB 00:08 2023-12-07T07:27:07.301 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:27:07.305 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:27:07.305 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:27:07.839 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:27:08.014 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86 1/1 2023-12-07T07:27:08.386 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:27:08.386 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:27:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86 1/1 2023-12-07T07:27:13.202 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86 1/1 2023-12-07T07:27:13.202 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:27:13.202 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:27:13.202 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:27:13.202 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:27:13.202 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:27:13.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test-debuginfo 2023-12-07T07:27:13.971 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:03:56 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:27:14.116 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:27:14.116 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 636 M 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 636 M 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.9 G 2023-12-07T07:27:14.117 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:27:26.776 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:27:26.939 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86 1/1 2023-12-07T07:27:31.723 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86 1/1 2023-12-07T07:27:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86 1/1 2023-12-07T07:27:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:27:32.017 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:27:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:27:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:27:32.017 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:27:32.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test-debuginfo 2023-12-07T07:27:32.807 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:59 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:27:32.952 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 636 M 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:27:32.953 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 636 M 2023-12-07T07:27:32.954 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.9 G 2023-12-07T07:27:32.954 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:27:42.148 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-test-debuginfo-19.0.0-64.ge068ebcf.el9.x86 23 MB/s | 636 MB 00:28 2023-12-07T07:27:42.149 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:27:42.149 INFO:teuthology.orchestra.run.smithi154.stdout:Total 23 MB/s | 636 MB 00:28 2023-12-07T07:27:42.149 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:27:42.155 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:27:42.155 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:27:45.899 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:27:45.900 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:28:04.074 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-test-debuginfo-19.0.0-64.ge068ebcf.el9.x86 20 MB/s | 636 MB 00:31 2023-12-07T07:28:04.074 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:28:04.075 INFO:teuthology.orchestra.run.smithi097.stdout:Total 20 MB/s | 636 MB 00:31 2023-12-07T07:28:04.075 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:28:04.080 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:28:04.080 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:28:07.840 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:28:07.840 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:28:52.588 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:28:52.687 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:02.299 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:02.560 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:02.560 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:02.560 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:29:02.560 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:29:02.560 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:02.561 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:29:02.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base-debuginfo 2023-12-07T07:29:03.419 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:05:46 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:29:03.514 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:29:03.564 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:29:03.565 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-12-07T07:29:03.565 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:29:03.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-debuginfo 2023-12-07T07:29:04.218 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:05:47 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:29:04.363 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 19 M 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:29:04.364 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:04.365 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 19 M 2023-12-07T07:29:04.365 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 102 M 2023-12-07T07:29:04.365 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:29:05.432 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-mgr-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 18 MB/s | 19 MB 00:01 2023-12-07T07:29:05.433 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:29:05.433 INFO:teuthology.orchestra.run.smithi154.stdout:Total 18 MB/s | 19 MB 00:01 2023-12-07T07:29:05.433 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:29:05.437 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:29:05.437 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:29:05.584 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:29:05.584 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:29:07.781 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:29:07.880 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:08.840 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:09.116 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:29:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:29:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:09.117 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:29:09.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mds-debuginfo 2023-12-07T07:29:09.903 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:05:52 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:29:10.048 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:29:10.049 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:10.049 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:29:10.049 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 48 M 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 48 M 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 173 M 2023-12-07T07:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:29:12.493 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-mds-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 20 MB/s | 48 MB 00:02 2023-12-07T07:29:12.493 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:29:12.494 INFO:teuthology.orchestra.run.smithi154.stdout:Total 19 MB/s | 48 MB 00:02 2023-12-07T07:29:12.494 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:29:12.498 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:29:12.498 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:29:12.809 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:29:12.810 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:29:14.638 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:29:14.984 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:17.525 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:29:17.615 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:18.974 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:19.255 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:19.255 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:19.255 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:29:19.255 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:29:19.255 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:19.256 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:29:19.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mon-debuginfo 2023-12-07T07:29:20.039 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:06:03 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:29:20.184 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 112 M 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 112 M 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 508 M 2023-12-07T07:29:20.185 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:29:22.525 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:22.795 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:22.795 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:22.795 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:29:22.795 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:29:22.795 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:22.795 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:29:23.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base-debuginfo 2023-12-07T07:29:23.612 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:05:50 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:29:23.707 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 is already installed. 2023-12-07T07:29:23.757 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:29:23.758 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2023-12-07T07:29:23.758 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:29:23.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-debuginfo 2023-12-07T07:29:24.412 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:05:51 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:29:24.556 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 19 M 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:24.557 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:29:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:24.558 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 19 M 2023-12-07T07:29:24.558 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 102 M 2023-12-07T07:29:24.558 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:29:25.177 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-mon-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 22 MB/s | 112 MB 00:04 2023-12-07T07:29:25.178 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:29:25.179 INFO:teuthology.orchestra.run.smithi154.stdout:Total 22 MB/s | 112 MB 00:04 2023-12-07T07:29:25.179 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:29:25.183 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:29:25.183 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:29:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-mgr-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 18 MB/s | 19 MB 00:01 2023-12-07T07:29:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:29:25.646 INFO:teuthology.orchestra.run.smithi097.stdout:Total 18 MB/s | 19 MB 00:01 2023-12-07T07:29:25.646 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:29:25.650 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:29:25.650 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:29:25.796 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:29:25.796 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:29:25.870 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:29:25.871 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:29:27.986 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:29:28.184 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:29.165 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:29.434 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:29.435 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:29:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:29:29.435 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:29.435 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:29:29.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mds-debuginfo 2023-12-07T07:29:30.203 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:05:57 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:29:30.348 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:29:30.348 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 48 M 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 48 M 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 173 M 2023-12-07T07:29:30.349 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:29:32.731 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-mds-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 20 MB/s | 48 MB 00:02 2023-12-07T07:29:32.732 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:29:32.732 INFO:teuthology.orchestra.run.smithi097.stdout:Total 20 MB/s | 48 MB 00:02 2023-12-07T07:29:32.732 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:29:32.736 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:29:32.736 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:29:33.049 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:29:33.050 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:29:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:29:37.665 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:37.788 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:29:37.947 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:39.291 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:39.567 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:39.568 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:39.568 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:29:39.568 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:29:39.568 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:39.568 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:29:39.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mon-debuginfo 2023-12-07T07:29:40.374 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:06:07 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:29:40.520 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:29:40.520 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 112 M 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:29:40.521 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 112 M 2023-12-07T07:29:40.522 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 508 M 2023-12-07T07:29:40.522 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:29:40.946 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:41.209 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:29:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:29:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:41.209 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:29:41.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-osd-debuginfo 2023-12-07T07:29:42.000 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:06:24 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:29:42.145 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:29:42.146 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:42.146 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:29:42.146 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:42.146 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:29:42.146 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 312 M 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 312 M 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 1.3 G 2023-12-07T07:29:42.147 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:29:46.542 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-mon-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 19 MB/s | 112 MB 00:06 2023-12-07T07:29:46.543 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:29:46.543 INFO:teuthology.orchestra.run.smithi097.stdout:Total 19 MB/s | 112 MB 00:06 2023-12-07T07:29:46.543 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:29:46.547 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:29:46.547 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:29:47.233 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:29:47.234 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:29:58.857 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:29:59.056 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:29:59.259 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-osd-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 18 MB/s | 312 MB 00:17 2023-12-07T07:29:59.260 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:29:59.260 INFO:teuthology.orchestra.run.smithi154.stdout:Total 18 MB/s | 312 MB 00:17 2023-12-07T07:29:59.261 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:29:59.265 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:29:59.265 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:30:01.122 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:30:01.123 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:30:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:02.630 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:02.630 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:30:02.630 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:30:02.630 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:30:02.630 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:30:02.630 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:30:02.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-osd-debuginfo 2023-12-07T07:30:03.416 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:06:30 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:30:03.559 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:30:03.560 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:30:03.560 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:30:03.560 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 312 M 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 312 M 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.3 G 2023-12-07T07:30:03.561 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:30:17.376 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-osd-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 23 MB/s | 312 MB 00:13 2023-12-07T07:30:17.436 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:30:17.436 INFO:teuthology.orchestra.run.smithi097.stdout:Total 23 MB/s | 312 MB 00:13 2023-12-07T07:30:17.436 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:30:17.436 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:30:17.436 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:30:19.238 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:30:19.238 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:30:33.220 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:30:33.320 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:41.326 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:41.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:41.600 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:41.600 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:30:41.600 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:30:41.600 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:41.600 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:30:41.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse-debuginfo 2023-12-07T07:30:42.411 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:07:25 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:30:42.556 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.9 M 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 5.9 M 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 28 M 2023-12-07T07:30:42.557 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:30:43.065 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-fuse-debuginfo-19.0.0-64.ge068ebcf.el9.x86 12 MB/s | 5.9 MB 00:00 2023-12-07T07:30:43.066 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:30:43.066 INFO:teuthology.orchestra.run.smithi154.stdout:Total 12 MB/s | 5.9 MB 00:00 2023-12-07T07:30:43.066 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:30:43.070 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:30:43.070 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:30:43.136 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:30:43.136 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:30:43.883 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:30:43.996 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:44.934 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:30:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:30:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:44.934 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:30:45.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel-debuginfo 2023-12-07T07:30:45.717 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:07:28 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:30:45.866 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:30:45.866 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:30:45.866 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:45.866 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 108 k 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 108 k 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 440 k 2023-12-07T07:30:45.867 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:30:46.177 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-debuginfo-19.0.0-64.ge068ebcf.el 348 kB/s | 108 kB 00:00 2023-12-07T07:30:46.177 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:30:46.177 INFO:teuthology.orchestra.run.smithi154.stdout:Total 347 kB/s | 108 kB 00:00 2023-12-07T07:30:46.177 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:30:46.181 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:30:46.181 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:30:46.214 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:30:46.214 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:30:46.326 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:30:46.424 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:30:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:30:47.065 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:30:47.066 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:47.066 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:30:47.066 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:30:47.066 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:47.066 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:30:47.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2-debuginfo 2023-12-07T07:30:47.863 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:07:30 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:30:48.008 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.2 M 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 5.2 M 2023-12-07T07:30:48.009 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 25 M 2023-12-07T07:30:48.010 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:30:48.496 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs2-debuginfo-19.0.0-64.ge068ebcf.el9.x8 11 MB/s | 5.2 MB 00:00 2023-12-07T07:30:48.496 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:30:48.497 INFO:teuthology.orchestra.run.smithi154.stdout:Total 11 MB/s | 5.2 MB 00:00 2023-12-07T07:30:48.497 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:30:48.500 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:30:48.500 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:30:48.565 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:30:48.565 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:30:49.221 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:30:49.310 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:30:49.801 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:30:50.097 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:30:50.097 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:50.097 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:30:50.097 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:30:50.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:50.098 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:30:50.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2-debuginfo 2023-12-07T07:30:50.861 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:07:33 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:30:51.007 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:30:51.007 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 44 M 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 44 M 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 251 M 2023-12-07T07:30:51.008 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:30:51.227 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:30:51.388 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:53.298 INFO:teuthology.orchestra.run.smithi154.stdout:librados2-debuginfo-19.0.0-64.ge068ebcf.el9.x86 19 MB/s | 44 MB 00:02 2023-12-07T07:30:53.299 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:30:53.299 INFO:teuthology.orchestra.run.smithi154.stdout:Total 19 MB/s | 44 MB 00:02 2023-12-07T07:30:53.300 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:30:53.304 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:30:53.304 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:30:53.604 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:30:53.604 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:30:58.583 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:30:58.680 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:59.641 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:59.905 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:30:59.905 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:30:59.905 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:30:59.905 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:30:59.905 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:30:59.905 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:00.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse-debuginfo 2023-12-07T07:31:00.552 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:00.682 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:07:27 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:00.826 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:00.827 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:00.827 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.9 M 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 5.9 M 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 28 M 2023-12-07T07:31:00.828 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:01.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1-debuginfo 2023-12-07T07:31:01.484 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-fuse-debuginfo-19.0.0-64.ge068ebcf.el9.x86 9.0 MB/s | 5.9 MB 00:00 2023-12-07T07:31:01.484 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:01.484 INFO:teuthology.orchestra.run.smithi097.stdout:Total 9.0 MB/s | 5.9 MB 00:00 2023-12-07T07:31:01.485 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:01.488 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:01.488 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:01.555 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:01.555 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:01.613 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:07:44 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:01.756 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 51 M 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:01.757 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 51 M 2023-12-07T07:31:01.758 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 303 M 2023-12-07T07:31:01.758 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:02.275 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:02.373 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:02.929 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:03.214 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:03.214 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:03.214 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:03.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel-debuginfo 2023-12-07T07:31:03.971 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:07:30 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:04.118 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 108 k 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:04.119 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:04.120 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:04.120 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 108 k 2023-12-07T07:31:04.120 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 440 k 2023-12-07T07:31:04.120 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:04.329 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-debuginfo-19.0.0-64.ge068ebcf.el 518 kB/s | 108 kB 00:00 2023-12-07T07:31:04.329 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:04.329 INFO:teuthology.orchestra.run.smithi097.stdout:Total 515 kB/s | 108 kB 00:00 2023-12-07T07:31:04.330 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:04.333 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:04.333 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:04.350 INFO:teuthology.orchestra.run.smithi154.stdout:librbd1-debuginfo-19.0.0-64.ge068ebcf.el9.x86_6 20 MB/s | 51 MB 00:02 2023-12-07T07:31:04.351 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:04.351 INFO:teuthology.orchestra.run.smithi154.stdout:Total 20 MB/s | 51 MB 00:02 2023-12-07T07:31:04.352 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:04.356 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:04.356 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:04.366 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:04.366 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:04.482 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:04.576 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:04.696 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:04.697 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:04.934 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:05.205 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:05.205 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:05.205 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:05.205 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:05.205 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:05.205 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:05.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2-debuginfo 2023-12-07T07:31:05.982 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:07:32 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:06.126 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 5.2 M 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:06.127 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:06.128 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:06.128 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 5.2 M 2023-12-07T07:31:06.128 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 25 M 2023-12-07T07:31:06.128 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:06.766 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs2-debuginfo-19.0.0-64.ge068ebcf.el9.x8 8.2 MB/s | 5.2 MB 00:00 2023-12-07T07:31:06.767 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:06.767 INFO:teuthology.orchestra.run.smithi097.stdout:Total 8.2 MB/s | 5.2 MB 00:00 2023-12-07T07:31:06.767 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:06.770 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:06.770 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:06.834 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:06.834 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:07.487 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:07.589 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:08.409 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:08.409 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:08.409 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:08.409 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:08.409 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:08.409 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:08.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2-debuginfo 2023-12-07T07:31:09.194 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:07:36 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:09.339 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 44 M 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:09.340 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:09.341 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:09.341 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:09.341 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 44 M 2023-12-07T07:31:09.341 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 251 M 2023-12-07T07:31:09.341 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:10.605 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:10.690 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:11.726 INFO:teuthology.orchestra.run.smithi097.stdout:librados2-debuginfo-19.0.0-64.ge068ebcf.el9.x86 18 MB/s | 44 MB 00:02 2023-12-07T07:31:11.727 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:11.728 INFO:teuthology.orchestra.run.smithi097.stdout:Total 18 MB/s | 44 MB 00:02 2023-12-07T07:31:11.728 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:11.732 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:11.732 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:12.030 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:12.030 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:13.163 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:13.163 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:13.163 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:13.163 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:13.163 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:13.163 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:13.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs-debuginfo 2023-12-07T07:31:13.949 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:07:56 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:14.094 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 615 k 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:14.095 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 615 k 2023-12-07T07:31:14.096 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.1 M 2023-12-07T07:31:14.096 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:14.519 INFO:teuthology.orchestra.run.smithi154.stdout:python3-cephfs-debuginfo-19.0.0-64.ge068ebcf.el 1.4 MB/s | 615 kB 00:00 2023-12-07T07:31:14.520 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:14.521 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.4 MB/s | 615 kB 00:00 2023-12-07T07:31:14.521 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:14.523 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:14.524 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:14.562 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:14.562 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:14.722 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:14.826 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:15.224 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:15.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:15.484 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:15.485 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:15.485 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:15.485 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:15.485 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:15.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados-debuginfo 2023-12-07T07:31:16.240 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:07:59 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:16.385 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:16.385 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:16.385 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:16.385 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 769 k 2023-12-07T07:31:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:16.386 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:16.386 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:16.386 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:16.386 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 769 k 2023-12-07T07:31:16.386 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.7 M 2023-12-07T07:31:16.387 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:16.814 INFO:teuthology.orchestra.run.smithi154.stdout:python3-rados-debuginfo-19.0.0-64.ge068ebcf.el9 1.8 MB/s | 769 kB 00:00 2023-12-07T07:31:16.814 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:16.817 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.8 MB/s | 769 kB 00:00 2023-12-07T07:31:16.817 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:16.818 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:16.818 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:16.855 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:16.856 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:17.031 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:17.037 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:17.133 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:17.137 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x8 1/1 2023-12-07T07:31:17.521 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x8 1/1 2023-12-07T07:31:17.799 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x8 1/1 2023-12-07T07:31:17.799 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:17.799 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:17.799 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:17.799 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:17.799 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:17.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd-debuginfo 2023-12-07T07:31:18.564 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:08:01 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:18.709 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.1 M 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 1.1 M 2023-12-07T07:31:18.710 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 4.3 M 2023-12-07T07:31:18.711 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:18.977 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:19.150 INFO:teuthology.orchestra.run.smithi154.stdout:python3-rbd-debuginfo-19.0.0-64.ge068ebcf.el9.x 2.6 MB/s | 1.1 MB 00:00 2023-12-07T07:31:19.151 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:19.151 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.6 MB/s | 1.1 MB 00:00 2023-12-07T07:31:19.151 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:19.154 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:19.154 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:19.196 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:19.196 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:19.262 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:19.263 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:19.263 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:19.263 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:19.263 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:19.263 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:19.409 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:19.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1-debuginfo 2023-12-07T07:31:19.525 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:19.924 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:20.051 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:07:47 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:20.196 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:20.197 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:20.197 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:20.197 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:20.197 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:20.197 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 51 M 2023-12-07T07:31:20.197 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:20.197 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:20.198 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:20.198 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:20.198 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:20.198 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 51 M 2023-12-07T07:31:20.198 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 303 M 2023-12-07T07:31:20.198 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:20.232 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:20.232 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:20.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw-debuginfo 2023-12-07T07:31:21.042 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:08:04 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:21.187 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 337 k 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:21.188 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 337 k 2023-12-07T07:31:21.189 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 1.0 M 2023-12-07T07:31:21.189 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:21.820 INFO:teuthology.orchestra.run.smithi154.stdout:python3-rgw-debuginfo-19.0.0-64.ge068ebcf.el9.x 535 kB/s | 337 kB 00:00 2023-12-07T07:31:21.820 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:21.820 INFO:teuthology.orchestra.run.smithi154.stdout:Total 533 kB/s | 337 kB 00:00 2023-12-07T07:31:21.821 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:21.824 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:21.824 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:21.861 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:21.862 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:21.994 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:22.109 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:22.491 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:22.755 INFO:teuthology.orchestra.run.smithi097.stdout:librbd1-debuginfo-19.0.0-64.ge068ebcf.el9.x86_6 20 MB/s | 51 MB 00:02 2023-12-07T07:31:22.756 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:22.756 INFO:teuthology.orchestra.run.smithi097.stdout:Total 20 MB/s | 51 MB 00:02 2023-12-07T07:31:22.756 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:22.760 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:22.760 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:22.780 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:22.780 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:22.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse-debuginfo 2023-12-07T07:31:23.104 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:23.104 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:23.567 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:08:06 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:23.712 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:23.712 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:23.712 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:23.712 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:23.712 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:23.713 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 507 k 2023-12-07T07:31:23.713 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:23.713 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:23.713 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:23.713 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:23.713 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:23.714 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 507 k 2023-12-07T07:31:23.714 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.5 M 2023-12-07T07:31:23.714 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:23.997 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 1.7 MB/s | 507 kB 00:00 2023-12-07T07:31:23.998 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:23.998 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.7 MB/s | 507 kB 00:00 2023-12-07T07:31:23.998 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:24.001 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:24.002 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:24.038 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:24.038 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:24.675 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:24.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:24.954 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:24.954 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:24.954 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:24.954 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:24.954 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:25.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror-debuginfo 2023-12-07T07:31:25.731 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:08:08 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:25.876 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:25.877 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:25.877 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:25.877 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:25.877 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:25.877 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 51 M 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 51 M 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 290 M 2023-12-07T07:31:25.878 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:28.317 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-debuginfo-19.0.0-64.ge068ebcf.el9.x8 21 MB/s | 51 MB 00:02 2023-12-07T07:31:28.318 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:28.318 INFO:teuthology.orchestra.run.smithi154.stdout:Total 21 MB/s | 51 MB 00:02 2023-12-07T07:31:28.318 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:28.322 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:28.322 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:28.654 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:28.654 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:29.023 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:29.128 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:31.270 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:31.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:31.562 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:31.562 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:31.562 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:31.562 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:31.562 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:31.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs-debuginfo 2023-12-07T07:31:32.342 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:07:59 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:32.488 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:32.488 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:32.488 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 615 k 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 615 k 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.1 M 2023-12-07T07:31:32.489 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:32.774 INFO:teuthology.orchestra.run.smithi097.stdout:python3-cephfs-debuginfo-19.0.0-64.ge068ebcf.el 2.1 MB/s | 615 kB 00:00 2023-12-07T07:31:32.774 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:32.775 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.1 MB/s | 615 kB 00:00 2023-12-07T07:31:32.775 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:32.778 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:32.778 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:32.814 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:32.814 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:32.979 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:33.098 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:33.486 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:33.744 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x 1/1 2023-12-07T07:31:33.744 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:33.744 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:33.745 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:33.745 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:33.745 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:33.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados-debuginfo 2023-12-07T07:31:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:34.488 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:08:01 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:34.540 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:34.635 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:34.636 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:34.636 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:34.636 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:34.636 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:34.636 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 769 k 2023-12-07T07:31:34.636 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:34.636 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:34.637 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:34.637 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:34.637 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:34.637 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 769 k 2023-12-07T07:31:34.637 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.7 M 2023-12-07T07:31:34.637 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:34.944 INFO:teuthology.orchestra.run.smithi097.stdout:python3-rados-debuginfo-19.0.0-64.ge068ebcf.el9 2.4 MB/s | 769 kB 00:00 2023-12-07T07:31:34.945 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:34.945 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.4 MB/s | 769 kB 00:00 2023-12-07T07:31:34.945 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:34.948 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:34.948 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:34.985 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:34.985 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:35.162 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:35.268 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x8 1/1 2023-12-07T07:31:35.695 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x8 1/1 2023-12-07T07:31:35.975 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x8 1/1 2023-12-07T07:31:35.976 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:35.976 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:35.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:35.976 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:35.976 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:36.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd-debuginfo 2023-12-07T07:31:36.576 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:36.741 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:08:03 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:36.886 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:36.887 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:36.887 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:36.887 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:36.887 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:36.887 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.1 M 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:36.888 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:36.889 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:36.889 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:36.889 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.1 M 2023-12-07T07:31:36.889 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 4.3 M 2023-12-07T07:31:36.889 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:37.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd-debuginfo 2023-12-07T07:31:37.364 INFO:teuthology.orchestra.run.smithi097.stdout:python3-rbd-debuginfo-19.0.0-64.ge068ebcf.el9.x 2.4 MB/s | 1.1 MB 00:00 2023-12-07T07:31:37.365 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:37.365 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.4 MB/s | 1.1 MB 00:00 2023-12-07T07:31:37.365 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:37.368 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:37.368 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:37.408 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:37.408 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:37.623 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:37.654 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:08:20 ago on Thu 07 Dec 2023 07:23:17 AM UTC. 2023-12-07T07:31:37.721 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:37.799 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-12-07T07:31:37.799 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.2 M 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-12-07T07:31:37.800 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:37.801 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 1.2 M 2023-12-07T07:31:37.801 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 5.5 M 2023-12-07T07:31:37.801 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-12-07T07:31:38.119 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-debuginfo-19.0.0-64.ge068ebcf.el9.x86_6 3.9 MB/s | 1.2 MB 00:00 2023-12-07T07:31:38.119 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:38.119 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.9 MB/s | 1.2 MB 00:00 2023-12-07T07:31:38.120 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-12-07T07:31:38.123 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-12-07T07:31:38.123 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-12-07T07:31:38.129 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:38.163 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-12-07T07:31:38.163 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-12-07T07:31:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-12-07T07:31:38.432 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:38.432 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:38.433 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:38.433 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:38.433 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:38.433 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:38.501 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:38.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw-debuginfo 2023-12-07T07:31:38.889 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:39.168 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:39.169 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:39.169 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-12-07T07:31:39.169 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:39.169 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:31:39.169 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-12-07T07:31:39.234 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:08:06 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:39.352 DEBUG:teuthology.parallel:result is None 2023-12-07T07:31:39.380 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:39.380 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:39.380 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:39.380 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:39.380 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:39.380 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 337 k 2023-12-07T07:31:39.381 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:39.381 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:39.381 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:39.381 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:39.381 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:39.381 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 337 k 2023-12-07T07:31:39.381 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.0 M 2023-12-07T07:31:39.382 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:39.643 INFO:teuthology.orchestra.run.smithi097.stdout:python3-rgw-debuginfo-19.0.0-64.ge068ebcf.el9.x 1.3 MB/s | 337 kB 00:00 2023-12-07T07:31:39.643 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:39.643 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.3 MB/s | 337 kB 00:00 2023-12-07T07:31:39.644 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:39.647 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:39.647 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:39.682 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:39.683 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:39.819 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:39.915 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:40.314 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:40.601 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_ 1/1 2023-12-07T07:31:40.601 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:40.601 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:40.601 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:40.601 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:40.601 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:40.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse-debuginfo 2023-12-07T07:31:41.402 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:08:08 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:41.546 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 507 k 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:41.547 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:41.548 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:41.548 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:41.548 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 507 k 2023-12-07T07:31:41.548 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.5 M 2023-12-07T07:31:41.548 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:41.979 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-debuginfo-19.0.0-64.ge068ebcf.el9.x86_ 1.1 MB/s | 507 kB 00:00 2023-12-07T07:31:41.980 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:41.980 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.1 MB/s | 507 kB 00:00 2023-12-07T07:31:41.980 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:41.983 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:41.983 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:42.020 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:42.021 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:42.181 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:42.294 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:42.951 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:42.951 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:42.952 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:42.952 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:42.952 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:42.952 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:43.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror-debuginfo 2023-12-07T07:31:43.745 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:08:10 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:43.890 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:43.891 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:43.891 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:43.891 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:43.891 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:43.891 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 51 M 2023-12-07T07:31:43.891 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:43.892 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:43.892 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:43.892 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:43.892 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:43.892 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 51 M 2023-12-07T07:31:43.892 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 290 M 2023-12-07T07:31:43.892 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:46.374 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-debuginfo-19.0.0-64.ge068ebcf.el9.x8 20 MB/s | 51 MB 00:02 2023-12-07T07:31:46.375 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:46.375 INFO:teuthology.orchestra.run.smithi097.stdout:Total 20 MB/s | 51 MB 00:02 2023-12-07T07:31:46.376 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:46.379 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:46.380 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:46.712 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:46.712 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:52.512 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:54.623 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:54.908 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_6 1/1 2023-12-07T07:31:54.908 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:54.909 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:54.909 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:54.909 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:54.909 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:55.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd-debuginfo 2023-12-07T07:31:55.711 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:08:22 ago on Thu 07 Dec 2023 07:23:33 AM UTC. 2023-12-07T07:31:55.858 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2023-12-07T07:31:55.858 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-64.ge068ebcf.el9 ceph 1.2 M 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.2 M 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 5.5 M 2023-12-07T07:31:55.859 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2023-12-07T07:31:56.168 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-debuginfo-19.0.0-64.ge068ebcf.el9.x86_6 4.0 MB/s | 1.2 MB 00:00 2023-12-07T07:31:56.169 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2023-12-07T07:31:56.169 INFO:teuthology.orchestra.run.smithi097.stdout:Total 4.0 MB/s | 1.2 MB 00:00 2023-12-07T07:31:56.169 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2023-12-07T07:31:56.172 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2023-12-07T07:31:56.172 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2023-12-07T07:31:56.212 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2023-12-07T07:31:56.212 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2023-12-07T07:31:56.446 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2023-12-07T07:31:56.542 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:56.984 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:57.273 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 1/1 2023-12-07T07:31:57.273 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:57.273 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2023-12-07T07:31:57.274 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-debuginfo-2:19.0.0-64.ge068ebcf.el9.x86_64 2023-12-07T07:31:57.274 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:31:57.274 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2023-12-07T07:31:57.466 DEBUG:teuthology.parallel:result is None 2023-12-07T07:31:57.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e068ebcf7ecc6503f24666fb6b152034d3fe1067 2023-12-07T07:31:57.625 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-07T07:31:57.645 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-64.ge068ebcf.el9 2023-12-07T07:31:57.645 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-64.ge068ebcf.el9 2023-12-07T07:31:57.646 INFO:teuthology.task.install:The correct ceph version 19.0.0-64.ge068ebcf is installed. 2023-12-07T07:31:57.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e068ebcf7ecc6503f24666fb6b152034d3fe1067 2023-12-07T07:31:57.793 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-07T07:31:57.818 INFO:teuthology.orchestra.run.smithi154.stdout:19.0.0-64.ge068ebcf.el9 2023-12-07T07:31:57.818 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-64.ge068ebcf.el9 2023-12-07T07:31:57.818 INFO:teuthology.task.install:The correct ceph version 19.0.0-64.ge068ebcf is installed. 2023-12-07T07:31:57.818 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-07T07:31:57.818 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:31:57.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-07T07:31:57.845 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:31:57.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-07T07:31:57.890 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-07T07:31:57.891 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:31:57.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2023-12-07T07:31:57.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-07T07:31:57.983 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:31:57.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2023-12-07T07:31:58.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-07T07:31:58.083 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-07T07:31:58.084 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:31:58.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-07T07:31:58.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-07T07:31:58.177 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:31:58.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-07T07:31:58.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-07T07:31:58.273 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-07T07:31:58.273 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:31:58.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2023-12-07T07:31:58.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-07T07:31:58.366 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:31:58.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2023-12-07T07:31:58.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-07T07:31:58.465 INFO:teuthology.run_tasks:Running task ceph... 2023-12-07T07:31:58.533 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-07T07:31:58.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /var/log/ceph 2023-12-07T07:31:58.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2023-12-07T07:31:58.563 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-07T07:31:58.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-07T07:31:58.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-07T07:31:58.631 INFO:tasks.ceph:Creating extra log directories... 2023-12-07T07:31:58.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-07T07:31:58.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-07T07:31:58.708 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-07T07:31:58.708 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-07T07:31:58.708 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571', 'branch': 'wip-yuri8-testing-2023-12-06-1425', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7481571', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri8-testing-2023-12-06-1425'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri8-testing-2023-12-06-1425', 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 4465, 'sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri8-testing-2023-12-06-1425', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'e068ebcf7ecc6503f24666fb6b152034d3fe1067', 'targets': {'smithi097.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB4RfLkzGipXJRP4cCG467MgQNHYm2xufGqX5/6siE8P+xhpBRivA6vqSff7pu0DIsxm5y8b8cm90uALQU0ZS+o=', 'smithi154.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeQXt1wvARrpJL3rnigyAmYqQUcE+qIhPN1th+Hbl2nIaVmXPX2zYdj7vZ1kMwMzNbFKgpVpWZumdUX91mLa0Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-07_01:25:59', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-07T07:31:58.709 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-07T07:31:58.737 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-07T07:31:58.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/run/ceph 2023-12-07T07:31:58.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2023-12-07T07:31:58.820 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:31:58.820 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2023-12-07T07:31:58.871 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-07T07:31:58.871 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2023-12-07T07:31:58.925 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-07 07:25:15.984594451 +0000 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-07 07:22:19.505166318 +0000 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-07 07:22:19.505166318 +0000 2023-12-07T07:31:58.926 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2023-12-07 07:22:19.505166318 +0000 2023-12-07T07:31:58.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-07T07:31:58.991 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-07T07:31:58.991 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-07T07:31:58.991 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000189969 s, 2.7 MB/s 2023-12-07T07:31:58.992 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-07T07:31:59.047 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2023-12-07T07:31:59.102 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-07 07:25:15.984594451 +0000 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-07 07:22:20.025156979 +0000 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-07 07:22:20.025156979 +0000 2023-12-07T07:31:59.103 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2023-12-07 07:22:20.025156979 +0000 2023-12-07T07:31:59.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-07T07:31:59.168 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-07T07:31:59.168 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-07T07:31:59.169 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000202028 s, 2.5 MB/s 2023-12-07T07:31:59.170 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-07T07:31:59.225 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2023-12-07T07:31:59.279 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-07 07:25:15.985594433 +0000 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-07 07:22:20.515148179 +0000 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-07 07:22:20.515148179 +0000 2023-12-07T07:31:59.280 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2023-12-07 07:22:20.515148179 +0000 2023-12-07T07:31:59.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-07T07:31:59.345 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-07T07:31:59.345 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-07T07:31:59.345 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000253042 s, 2.0 MB/s 2023-12-07T07:31:59.346 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-07T07:31:59.401 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-07 07:25:15.985594433 +0000 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-07 07:22:20.997139522 +0000 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-07 07:22:20.997139522 +0000 2023-12-07T07:31:59.457 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2023-12-07 07:22:20.997139522 +0000 2023-12-07T07:31:59.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-07T07:31:59.522 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-07T07:31:59.522 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-07T07:31:59.522 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000196777 s, 2.6 MB/s 2023-12-07T07:31:59.523 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-07T07:31:59.579 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T07:31:59.579 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:31:59.580 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2023-12-07T07:31:59.595 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-07T07:31:59.595 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2023-12-07T07:31:59.649 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-12-07 07:25:00.855267042 +0000 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-12-07 07:22:19.503623008 +0000 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-12-07 07:22:19.503623008 +0000 2023-12-07T07:31:59.650 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2023-12-07 07:22:19.503623008 +0000 2023-12-07T07:31:59.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-07T07:31:59.715 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-12-07T07:31:59.715 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-12-07T07:31:59.715 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000206327 s, 2.5 MB/s 2023-12-07T07:31:59.716 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-07T07:31:59.772 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2023-12-07T07:31:59.827 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-07T07:31:59.827 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:31:59.828 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-07T07:31:59.828 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:31:59.828 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:31:59.828 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-12-07 07:25:00.856267017 +0000 2023-12-07T07:31:59.828 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-12-07 07:22:20.016609523 +0000 2023-12-07T07:31:59.828 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-12-07 07:22:20.016609523 +0000 2023-12-07T07:31:59.828 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2023-12-07 07:22:20.016609523 +0000 2023-12-07T07:31:59.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-07T07:31:59.893 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-12-07T07:31:59.893 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-12-07T07:31:59.893 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000191578 s, 2.7 MB/s 2023-12-07T07:31:59.894 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-07T07:31:59.951 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2023-12-07T07:32:00.005 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-07T07:32:00.005 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:32:00.006 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-07T07:32:00.006 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:32:00.006 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:32:00.006 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-12-07 07:25:00.856267017 +0000 2023-12-07T07:32:00.006 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-12-07 07:22:20.502596748 +0000 2023-12-07T07:32:00.006 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-12-07 07:22:20.502596748 +0000 2023-12-07T07:32:00.006 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2023-12-07 07:22:20.502596748 +0000 2023-12-07T07:32:00.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-07T07:32:00.071 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-12-07T07:32:00.071 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-12-07T07:32:00.071 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000175443 s, 2.9 MB/s 2023-12-07T07:32:00.072 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-07T07:32:00.128 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-12-07 07:25:00.856267017 +0000 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-12-07 07:22:20.990583921 +0000 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-12-07 07:22:20.990583921 +0000 2023-12-07T07:32:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2023-12-07 07:22:20.990583921 +0000 2023-12-07T07:32:00.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-07T07:32:00.249 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-12-07T07:32:00.249 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-12-07T07:32:00.249 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.00023453 s, 2.2 MB/s 2023-12-07T07:32:00.250 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-07T07:32:00.306 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:00.306 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-07T07:32:00.307 INFO:tasks.ceph:Generating config... 2023-12-07T07:32:00.308 INFO:tasks.ceph:[client] debug monc = 20 2023-12-07T07:32:00.308 INFO:tasks.ceph:[client] debug ms = 1 2023-12-07T07:32:00.308 INFO:tasks.ceph:[client] debug objecter = 20 2023-12-07T07:32:00.308 INFO:tasks.ceph:[client] debug rados = 20 2023-12-07T07:32:00.308 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-07T07:32:00.308 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-12-07T07:32:00.308 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-07T07:32:00.308 INFO:tasks.ceph:[global] ms type = async 2023-12-07T07:32:00.308 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-12-07T07:32:00.308 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-07T07:32:00.308 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-07T07:32:00.308 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-07T07:32:00.308 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-12-07T07:32:00.309 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-07T07:32:00.309 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-07T07:32:00.309 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-07T07:32:00.309 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-07T07:32:00.309 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-07T07:32:00.309 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-07T07:32:00.309 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-12-07T07:32:00.309 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] osd class default list = * 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] osd class load list = * 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-07T07:32:00.310 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-07T07:32:00.311 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-12-07T07:32:00.311 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-12-07T07:32:00.311 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-12-07T07:32:00.311 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-07T07:32:00.311 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-07T07:32:00.311 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-07T07:32:00.311 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-07T07:32:00.311 INFO:tasks.ceph:Setting up mon.a... 2023-12-07T07:32:00.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-07T07:32:00.358 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.keyring 2023-12-07T07:32:00.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-07T07:32:00.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T07:32:00.472 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.97'), ('mon.c', '[v2:172.21.15.97:3301,v1:172.21.15.97:6790]'), ('mon.b', '172.21.15.154')] 2023-12-07T07:32:00.473 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.97,[v2:172.21.15.97:3301,v1:172.21.15.97:6790],172.21.15.154', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-07T07:32:00.474 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:00.474 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-07T07:32:00.528 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.97 --addv c '[v2:172.21.15.97:3301,v1:172.21.15.97:6790]' --add b 172.21.15.154 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:00.613 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:00.613 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: generated fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = pacific 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 0 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:min_mon_release 16 (pacific) 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:election_strategy: 1 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:00.614 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-07T07:32:00.616 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-07T07:32:00.630 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f040f620-331a-4806-a30a-a56e2cce06a5... 2023-12-07T07:32:00.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-07T07:32:00.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-07T07:32:00.720 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2023-12-07T07:32:00.721 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2023-12-07T07:32:00.722 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2023-12-07T07:32:00.723 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.97,[v2:172.21.15.97:3301,v1:172.21.15.97:6790],172.21.15.154" 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: mon client directed command retry = 5 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: mon election default strategy = 1 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: ms inject socket failures = 5000 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: ms type = async 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: osd heartbeat grace = 80 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2023-12-07T07:32:00.724 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2023-12-07T07:32:00.725 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2023-12-07T07:32:00.726 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore zero block detection = True 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: debug monc = 20 2023-12-07T07:32:00.727 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 10 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd class default list = * 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd class load list = * 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd client watch timeout = 120 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify cached snaps = True 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify missing on start = True 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2023-12-07T07:32:00.728 INFO:teuthology.orchestra.run.smithi097.stdout: osd fast shutdown = False 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock override recovery settings = True 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd op queue = debug_random 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd op queue cut off = debug_random 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: osd shutdown pgref assert = True 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2023-12-07T07:32:00.729 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2023-12-07T07:32:00.730 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2023-12-07T07:32:00.731 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon min osdmap epochs = 50 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd crush smoke test = False 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune interval = 2 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune min = 15 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune txsize = 2 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon scrub interval = 300 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on pool no app = False 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: paxos service trim min = 10 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2023-12-07T07:32:00.732 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: debug monc = 20 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: debug objecter = 20 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout: debug rados = 20 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2023-12-07T07:32:00.733 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2023-12-07T07:32:00.734 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2023-12-07T07:32:00.734 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2023-12-07T07:32:00.734 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-07T07:32:00.734 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2023-12-07T07:32:00.734 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.734 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-07T07:32:00.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.736 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = "172.21.15.97,[v2:172.21.15.97:3301,v1:172.21.15.97:6790],172.21.15.154" 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: mon client directed command retry = 5 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: mon election default strategy = 1 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject socket failures = 5000 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: ms type = async 2023-12-07T07:32:00.737 INFO:teuthology.orchestra.run.smithi154.stdout: osd heartbeat grace = 80 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock profile = high_recovery_ops 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2023-12-07T07:32:00.738 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug trim objects = true 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.739 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: bdev async discard = True 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: bdev enable discard = True 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore allocator = bitmap 2023-12-07T07:32:00.740 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore block size = 96636764160 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore fsck on mount = True 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore zero block detection = True 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluefs = 1 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluestore = 1 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: debug monc = 20 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 20 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: debug rocksdb = 10 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd backfillfull_ratio = 0.85 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd full ratio = 0.9 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd nearfull ratio = 0.8 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: osd class default list = * 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: osd class load list = * 2023-12-07T07:32:00.741 INFO:teuthology.orchestra.run.smithi154.stdout: osd client watch timeout = 120 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify cached snaps = True 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify missing on start = True 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd failsafe full ratio = 0.95 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd fast shutdown = False 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock override recovery settings = True 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd objectstore = bluestore 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue = debug_random 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue cut off = debug_random 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: osd shutdown pgref assert = True 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2023-12-07T07:32:00.742 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2023-12-07T07:32:00.743 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: auth mon ticket ttl = 660 # 11m 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: auth service ticket ttl = 240 # 4m 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: # don't complain about insecure global_id in the test suite 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: # 1m isn't quite enough 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: mon_down_mkfs_grace = 2m 2023-12-07T07:32:00.744 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_filestore_osds = false 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon min osdmap epochs = 50 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd crush smoke test = False 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune interval = 2 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune min = 15 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune txsize = 2 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon scrub interval = 300 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on pool no app = False 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: paxos service trim min = 10 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-07T07:32:00.745 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-07T07:32:00.746 INFO:teuthology.orchestra.run.smithi154.stdout: debug monc = 20 2023-12-07T07:32:00.746 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-12-07T07:32:00.746 INFO:teuthology.orchestra.run.smithi154.stdout: debug objecter = 20 2023-12-07T07:32:00.746 INFO:teuthology.orchestra.run.smithi154.stdout: debug rados = 20 2023-12-07T07:32:00.746 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2023-12-07T07:32:00.746 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.c] 2023-12-07T07:32:00.746 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.b] 2023-12-07T07:32:00.748 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-07T07:32:00.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-07T07:32:00.842 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-07T07:32:00.842 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:00.842 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-07T07:32:00.857 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:00.857 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-07T07:32:00.911 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T07:32:00.912 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:00.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-07T07:32:00.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T07:32:00.991 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:00.991 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:01.046 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T07:32:01.046 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:01.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-07T07:32:01.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T07:32:01.086 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:01.086 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:01.140 INFO:tasks.ceph:Setting up mon nodes... 2023-12-07T07:32:01.140 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-07T07:32:01.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-07T07:32:01.194 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-07T07:32:01.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-07T07:32:01.255 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-07T07:32:01.258 INFO:tasks.ceph:Setting up mds nodes... 2023-12-07T07:32:01.258 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-07T07:32:01.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-07T07:32:01.300 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-07T07:32:01.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-07T07:32:01.357 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-07T07:32:01.372 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-07T07:32:01.373 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-07T07:32:01.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-07T07:32:01.399 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:01.399 INFO:tasks.ceph:role: osd.0 2023-12-07T07:32:01.399 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T07:32:01.399 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:01.473 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:01.474 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:01.953 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-07T07:32:01.955 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-07T07:32:01.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-07T07:32:02.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-07T07:32:02.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-07T07:32:02.193 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:02.193 INFO:tasks.ceph:role: osd.1 2023-12-07T07:32:02.193 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T07:32:02.193 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:02.267 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:02.268 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:02.268 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:02.731 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-07T07:32:02.733 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-07T07:32:02.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-07T07:32:02.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-07T07:32:02.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-07T07:32:02.926 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:02.926 INFO:tasks.ceph:role: osd.2 2023-12-07T07:32:02.926 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T07:32:02.927 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-07T07:32:03.001 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:03.001 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:03.001 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:03.001 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:03.001 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:03.001 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:03.002 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:03.002 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:03.002 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:03.002 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:03.484 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-07T07:32:03.485 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-07T07:32:03.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-07T07:32:03.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-07T07:32:03.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-07T07:32:03.735 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:03.735 INFO:tasks.ceph:role: osd.3 2023-12-07T07:32:03.735 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T07:32:03.736 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-07T07:32:03.810 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:03.810 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:03.811 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:03.811 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:03.811 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:03.811 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:03.811 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:04.329 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-07T07:32:04.332 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-07T07:32:04.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-07T07:32:04.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-07T07:32:04.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:04.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:04.610+0000 7f8e468be500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-07T07:32:04.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:04.610+0000 7f8e468be500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-07T07:32:04.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:04.610+0000 7f8e468be500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T07:32:04.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:04.610+0000 7f8e468be500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T07:32:04.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:04.610+0000 7f8e468be500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T07:32:04.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:04.610+0000 7f8e468be500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-07T07:32:07.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-07T07:32:07.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:07.574+0000 7f171a249500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-07T07:32:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:07.575+0000 7f171a249500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-07T07:32:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:07.575+0000 7f171a249500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T07:32:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:07.575+0000 7f171a249500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T07:32:07.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:07.575+0000 7f171a249500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T07:32:07.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:07.575+0000 7f171a249500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-07T07:32:10.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-07T07:32:10.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:10.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:10.539+0000 7fa27460d500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-07T07:32:10.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:10.540+0000 7fa27460d500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-07T07:32:10.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:10.540+0000 7fa27460d500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T07:32:10.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:10.540+0000 7fa27460d500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T07:32:10.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:10.540+0000 7fa27460d500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T07:32:10.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:10.540+0000 7fa27460d500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-07T07:32:13.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-07T07:32:13.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:13.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:13.509+0000 7f58dd414500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-07T07:32:13.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:13.509+0000 7f58dd414500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-07T07:32:13.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:13.509+0000 7f58dd414500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-07T07:32:13.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:13.509+0000 7f58dd414500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-07T07:32:13.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:13.509+0000 7f58dd414500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-07T07:32:13.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:13.510+0000 7f58dd414500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-07T07:32:16.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-07T07:32:16.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-07T07:32:16.423 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:16.423 INFO:tasks.ceph:role: osd.4 2023-12-07T07:32:16.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T07:32:16.424 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-07T07:32:16.501 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:16.501 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:16.501 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:16.501 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:16.502 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:16.502 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:16.502 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:16.502 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:16.502 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:16.502 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:17.006 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-12-07T07:32:17.008 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-12-07T07:32:17.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-07T07:32:17.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-07T07:32:17.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-07T07:32:17.239 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:17.239 INFO:tasks.ceph:role: osd.5 2023-12-07T07:32:17.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T07:32:17.239 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:17.314 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:17.315 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:17.315 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:17.773 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-12-07T07:32:17.775 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-12-07T07:32:17.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-07T07:32:17.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-07T07:32:17.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-07T07:32:17.978 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:17.979 INFO:tasks.ceph:role: osd.6 2023-12-07T07:32:17.979 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T07:32:17.979 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:18.055 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:18.536 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-12-07T07:32:18.539 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-12-07T07:32:18.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-07T07:32:18.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-07T07:32:18.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-07T07:32:18.776 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-07T07:32:18.776 INFO:tasks.ceph:role: osd.7 2023-12-07T07:32:18.776 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T07:32:18.776 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-07T07:32:18.850 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T07:32:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T07:32:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T07:32:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-07T07:32:18.851 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T07:32:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-12-07T07:32:18.851 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T07:32:18.851 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-07T07:32:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T07:32:18.851 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T07:32:19.373 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-12-07T07:32:19.375 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-12-07T07:32:19.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-07T07:32:19.520 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-07T07:32:19.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:19.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:19.650+0000 7f6c2d81a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-07T07:32:19.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:19.651+0000 7f6c2d81a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-07T07:32:19.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:19.651+0000 7f6c2d81a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-07T07:32:19.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:19.651+0000 7f6c2d81a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-07T07:32:19.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:19.651+0000 7f6c2d81a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-07T07:32:19.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:19.651+0000 7f6c2d81a500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-07T07:32:22.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-07T07:32:22.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:22.623 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:22.621+0000 7fcef28af500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-07T07:32:22.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:22.621+0000 7fcef28af500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-07T07:32:22.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:22.621+0000 7fcef28af500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-07T07:32:22.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:22.621+0000 7fcef28af500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-07T07:32:22.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:22.621+0000 7fcef28af500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-07T07:32:22.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:22.622+0000 7fcef28af500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-07T07:32:25.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-07T07:32:25.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:25.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:25.580+0000 7fd5f2ca3500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-07T07:32:25.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:25.580+0000 7fd5f2ca3500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-07T07:32:25.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:25.580+0000 7fd5f2ca3500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-07T07:32:25.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:25.580+0000 7fd5f2ca3500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-07T07:32:25.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:25.580+0000 7fd5f2ca3500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-07T07:32:25.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:25.581+0000 7fd5f2ca3500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-07T07:32:28.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-07T07:32:28.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:28.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:28.554+0000 7f27ae99b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-07T07:32:28.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:28.554+0000 7f27ae99b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-07T07:32:28.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:28.554+0000 7f27ae99b500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-07T07:32:28.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:28.554+0000 7f27ae99b500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-07T07:32:28.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:28.554+0000 7f27ae99b500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-07T07:32:28.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:28.555+0000 7f27ae99b500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-07T07:32:31.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-07T07:32:31.427 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-07T07:32:31.427 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:31.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-07T07:32:31.459 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:31.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-07T07:32:31.529 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:31.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-07T07:32:31.601 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:31.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-07T07:32:31.670 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:31.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-07T07:32:31.738 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:31.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-07T07:32:31.769 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:31.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-07T07:32:31.840 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:31.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-07T07:32:31.910 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:31.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-07T07:32:31.979 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:31.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-07T07:32:32.048 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:32.048 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-07T07:32:32.065 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:32.065 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-07T07:32:32.104 INFO:tasks.ceph:Adding keys to all mons... 2023-12-07T07:32:32.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-07T07:32:32.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.y] 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDxdHFl/8KPCxAA1RfcW/6R4g1SegMvXLlGiQ== 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQD0dHFlUf5sJBAA0vo79e+3fR4lsGz2nuE1Ww== 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQD3dHFlp5hVIhAA4DCh5nN2QkqW4HNOIyHtxA== 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQD6dHFlGaA8IBAARniFd6LJL6MdCYU7K/Nc+Q== 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.3] 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQD9dHFlBeFvHhAAhKuq5liPX9o8ZTpSzwZyzg== 2023-12-07T07:32:32.148 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.x] 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDxdHFl0cUpDxAAEXVEnXTFmujonGXUEexc9g== 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.4] 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQADdXFl4wXZJhAAGm46CmH8gJ64ZAgWnDpOXw== 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.5] 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQAGdXFlFE0aJRAAy0EeVsZ8eDM1wtwzYzJdVA== 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.6] 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQAJdXFlWgmpIhAA0ZjoA/tyGLr59FJMwIxM2w== 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.7] 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQAMdXFlTQ4bIRAAtRN8NaVF03YuFohv72cmvw== 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDxdHFlkjvTERAAuEIfvHQJoBxRYab/xUQJoA== 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout:[client.1] 2023-12-07T07:32:32.149 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDxdHFlaoc6FRAAU2KB5GBqgFf9hkEt3AVQng== 2023-12-07T07:32:32.170 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.y] 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDxdHFl/8KPCxAA1RfcW/6R4g1SegMvXLlGiQ== 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD0dHFlUf5sJBAA0vo79e+3fR4lsGz2nuE1Ww== 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD3dHFlp5hVIhAA4DCh5nN2QkqW4HNOIyHtxA== 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD6dHFlGaA8IBAARniFd6LJL6MdCYU7K/Nc+Q== 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.3] 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD9dHFlBeFvHhAAhKuq5liPX9o8ZTpSzwZyzg== 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDxdHFl0cUpDxAAEXVEnXTFmujonGXUEexc9g== 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.4] 2023-12-07T07:32:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQADdXFl4wXZJhAAGm46CmH8gJ64ZAgWnDpOXw== 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.5] 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAGdXFlFE0aJRAAy0EeVsZ8eDM1wtwzYzJdVA== 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.6] 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAJdXFlWgmpIhAA0ZjoA/tyGLr59FJMwIxM2w== 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.7] 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAMdXFlTQ4bIRAAtRN8NaVF03YuFohv72cmvw== 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDxdHFlkjvTERAAuEIfvHQJoBxRYab/xUQJoA== 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout:[client.1] 2023-12-07T07:32:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDxdHFlaoc6FRAAU2KB5GBqgFf9hkEt3AVQng== 2023-12-07T07:32:32.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T07:32:32.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T07:32:32.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T07:32:32.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T07:32:32.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T07:32:32.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T07:32:32.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T07:32:32.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T07:32:32.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T07:32:32.865 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-07T07:32:32.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-07T07:32:32.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-07T07:32:32.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-07T07:32:33.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-07T07:32:33.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-07T07:32:33.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-07T07:32:33.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-07T07:32:33.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-07T07:32:33.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-07T07:32:33.358 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:33.361 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-07T07:32:33.413 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-07T07:32:33.414 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-07T07:32:33.414 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-07T07:32:33.414 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-07T07:32:33.416 INFO:tasks.ceph.mon.a:Started 2023-12-07T07:32:33.416 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-07T07:32:33.416 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-07T07:32:33.417 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-07T07:32:33.418 INFO:tasks.ceph.mon.c:Started 2023-12-07T07:32:33.418 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-07T07:32:33.419 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-07T07:32:33.419 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-07T07:32:33.456 INFO:tasks.ceph.mon.b:Started 2023-12-07T07:32:33.456 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-07T07:32:33.456 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-07T07:32:33.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-07T07:32:33.459 INFO:tasks.ceph.mgr.y:Started 2023-12-07T07:32:33.459 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-07T07:32:33.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-07T07:32:33.461 INFO:tasks.ceph.mgr.x:Started 2023-12-07T07:32:33.461 INFO:tasks.ceph:Setting crush tunables to default 2023-12-07T07:32:33.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd crush tunables default 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.562+0000 7ff17feb4640 10 monclient: get_monmap_and_config 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.562+0000 7ff17feb4640 10 monclient: build_initial_monmap 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.562+0000 7ff17feb4640 10 monclient: monmap: 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:33.563892+0000 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:33.563892+0000 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:33.566 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:33.566 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:33.566 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:33.566 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:33.566 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:33.566 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:33.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.563+0000 7ff17feb4640 1 Processor -- start 2023-12-07T07:32:33.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.563+0000 7ff17feb4640 1 -- start start 2023-12-07T07:32:33.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.563+0000 7ff17feb4640 10 monclient: init 2023-12-07T07:32:33.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:33.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1780a8a00 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 0x7ff1780a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient(hunting): picked mon.noname-c con 0x7ff1780b2260 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 0x7ff17813f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient(hunting): picked mon.noname-b con 0x7ff17813f080 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1780aa970 con 0x7ff1780a8a00 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1781421b0 con 0x7ff1780b2260 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780aac50 con 0x7ff17813f080 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17dc50640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff1780a8a00 legacy=0x7ff1781425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:33.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17feb4640 10 monclient(hunting): authenticate will time out at 1275.942636s 2023-12-07T07:32:33.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17e451640 1 -- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 msgr2=0x7ff17813f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3301/0 2023-12-07T07:32:33.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17d44f640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 msgr2=0x7ff1780a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2023-12-07T07:32:33.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17d44f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 0x7ff1780a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:33.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.564+0000 7ff17e451640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 0x7ff17813f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:33.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.765+0000 7ff17dc50640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff1780a8a00 legacy=0x7ff1781425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:33.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.765+0000 7ff17e451640 1 -- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 msgr2=0x7ff17813f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3301/0 2023-12-07T07:32:33.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.765+0000 7ff17e451640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 0x7ff17813f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:33.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.765+0000 7ff17d44f640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 msgr2=0x7ff1780a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2023-12-07T07:32:33.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:33.765+0000 7ff17d44f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 0x7ff1780a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:34.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.166+0000 7ff17dc50640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff1780a8a00 legacy=0x7ff1781425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:34.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.166+0000 7ff17e451640 1 -- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 msgr2=0x7ff17813f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3301/0 2023-12-07T07:32:34.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.166+0000 7ff17d44f640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 msgr2=0x7ff1780a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2023-12-07T07:32:34.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.166+0000 7ff17e451640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 0x7ff17813f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T07:32:34.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.166+0000 7ff17d44f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 0x7ff1780a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T07:32:34.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.967+0000 7ff17dc50640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff1780a8a00 legacy=0x7ff1781425d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:34.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.967+0000 7ff17e451640 1 -- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 msgr2=0x7ff17813f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3301/0 2023-12-07T07:32:34.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.967+0000 7ff17e451640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 0x7ff17813f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T07:32:34.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.967+0000 7ff17d44f640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 msgr2=0x7ff1780a8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2023-12-07T07:32:34.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:34.967+0000 7ff17d44f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 0x7ff1780a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T07:32:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient(hunting): tick 2023-12-07T07:32:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient(hunting): _check_auth_tickets 2023-12-07T07:32:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 monclient(hunting): continuing hunt 2023-12-07T07:32:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T07:32:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 -- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 msgr2=0x7ff17813f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:36.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff17813f080 0x7ff17813f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:36.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 msgr2=0x7ff1780a8de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:36.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1780b2260 0x7ff1780a8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:36.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff1780a8a00 legacy=0x7ff1781425d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:36.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient: _add_conns ranks=[3,1] 2023-12-07T07:32:36.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 0x7ff15c005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:36.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff15c0036c0 addr v2:172.21.15.97:3300/0 2023-12-07T07:32:36.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff15c006230 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:36.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1780aaf30 con 0x7ff15c006230 2023-12-07T07:32:36.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:36.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1781421b0 con 0x7ff15c0036c0 2023-12-07T07:32:36.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff16f7fe640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:36.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:36.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:36.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.564+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c006230 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:36.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.765+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:36.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.765+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:36.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:36.765+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c006230 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:37.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:37.166+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:37.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:37.166+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T07:32:37.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:37.166+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c006230 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:37.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:37.967+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:37.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:37.967+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T07:32:37.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:37.967+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c006230 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:39.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): tick 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): _check_auth_tickets 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 monclient(hunting): continuing hunt 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c0036c0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c006230 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff17e451640 1 -- reap_dead start 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 0x7ff15c005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff15c008ec0 addr v2:172.21.15.97:3300/0 2023-12-07T07:32:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff15c009340 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff15c009870 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1780aa970 con 0x7ff15c009870 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1780aaf30 con 0x7ff15c009340 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1781421b0 con 0x7ff15c008ec0 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff16f7fe640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff17e451640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff15c009870 legacy=0x7ff15c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:39.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.564+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c009340 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:39.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.765+0000 7ff17e451640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff15c009870 legacy=0x7ff15c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:39.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.765+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:39.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.765+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:39.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:39.765+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c009340 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:40.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.166+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:40.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.166+0000 7ff17e451640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff15c009870 legacy=0x7ff15c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:40.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.166+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T07:32:40.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.166+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c009340 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:40.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.967+0000 7ff17dc50640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3300/0 2023-12-07T07:32:40.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.967+0000 7ff17e451640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff15c009870 legacy=0x7ff15c00d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2023-12-07T07:32:40.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.967+0000 7ff17dc50640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T07:32:40.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:40.967+0000 7ff17d44f640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c009340 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2023-12-07T07:32:41.053 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:00:07.490 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:32:41.220 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:00:07.669 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:32:41.221 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:00:07.671 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:32:41.364 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:00:07.802 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:32:41.546 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:00:07.994 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:32:41.547 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:00:07.996 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:32:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): tick 2023-12-07T07:32:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): _check_auth_tickets 2023-12-07T07:32:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 monclient(hunting): continuing hunt 2023-12-07T07:32:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T07:32:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 -- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 msgr2=0x7ff15c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff15c008ec0 0x7ff15c005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ff15c009340 legacy=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff15c009870 legacy=0x7ff15c00d170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient: _add_conns ranks=[4,2] 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff15c00e8c0 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7ff15c00f340 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1781421b0 con 0x7ff15c00e8c0 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780aac50 con 0x7ff15c00f340 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff16f7fe640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff17d44f640 1 -- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 msgr2=0x7ff15c00f750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3301/0 2023-12-07T07:32:42.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.564+0000 7ff17d44f640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:42.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.765+0000 7ff17d44f640 1 -- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 msgr2=0x7ff15c00f750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.97:3301/0 2023-12-07T07:32:42.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.765+0000 7ff17d44f640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:42.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.830+0000 7ff17dc50640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:42.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.859+0000 7ff17dc50640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:43888/0 (socket says 172.21.15.97:43888) 2023-12-07T07:32:42.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.859+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 learned_addr learned my addr 172.21.15.97:0/4179546157 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:32:42.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.859+0000 7ff17dc50640 10 monclient(hunting): get_auth_request con 0x7ff15c00e8c0 auth_method 0 2023-12-07T07:32:42.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.859+0000 7ff17dc50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:42.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.859+0000 7ff17dc50640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:42.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.859+0000 7ff17dc50640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:42.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.916+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T07:32:42.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.916+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:42.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.916+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:42.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:42.916+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:43.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.117+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:43.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.117+0000 7ff17dc50640 10 monclient(hunting): get_auth_request con 0x7ff15c00e8c0 auth_method 0 2023-12-07T07:32:43.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.117+0000 7ff17dc50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:43.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.117+0000 7ff17dc50640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:43.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.117+0000 7ff17dc50640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:43.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.118+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T07:32:43.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.118+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:43.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.118+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:43.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.118+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:43.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.288+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.289+0000 7ff17d44f640 10 monclient(hunting): get_auth_request con 0x7ff15c00f340 auth_method 0 2023-12-07T07:32:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.289+0000 7ff17d44f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.289+0000 7ff17d44f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:43.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.289+0000 7ff17d44f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:43.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.292+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 msgr2=0x7ff15c00f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T07:32:43.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.292+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 msgr2=0x7ff15c00f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:43.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.292+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:43.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.292+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T07:32:43.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.585+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:43.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.586+0000 7ff17dc50640 10 monclient(hunting): get_auth_request con 0x7ff15c00e8c0 auth_method 0 2023-12-07T07:32:43.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.586+0000 7ff17dc50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:43.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.586+0000 7ff17dc50640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:43.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.586+0000 7ff17dc50640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:43.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.587+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T07:32:43.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.587+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:43.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.587+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:43.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:43.587+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-07T07:32:44.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.114+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:44.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.114+0000 7ff17d44f640 10 monclient(hunting): get_auth_request con 0x7ff15c00f340 auth_method 0 2023-12-07T07:32:44.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.114+0000 7ff17d44f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:44.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.114+0000 7ff17d44f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:44.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.114+0000 7ff17d44f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:44.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.115+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 msgr2=0x7ff15c00f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T07:32:44.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.115+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 msgr2=0x7ff15c00f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:44.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.115+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:44.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.115+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T07:32:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.389+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:44.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.389+0000 7ff17dc50640 10 monclient(hunting): get_auth_request con 0x7ff15c00e8c0 auth_method 0 2023-12-07T07:32:44.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.389+0000 7ff17dc50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:44.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.389+0000 7ff17dc50640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:44.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.389+0000 7ff17dc50640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:44.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.390+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-07T07:32:44.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.390+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:44.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.390+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:44.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:44.390+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 10 monclient(hunting): tick 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 10 monclient(hunting): _check_auth_tickets 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 1 monclient(hunting): continuing hunt 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 10 monclient(hunting): _reopen_session rank -1 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 msgr2=0x7ff15c00f750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c00f340 0x7ff15c00f750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 msgr2=0x7ff15c00ecd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.564+0000 7ff16f7fe640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00e8c0 0x7ff15c00ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff17e451640 1 -- 172.21.15.97:0/4179546157 reap_dead start 2023-12-07T07:32:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff15c009870 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff15c00eca0 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7ff15c005b40 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 1 -- 172.21.15.97:0/4179546157 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c0108b0 con 0x7ff15c009870 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 1 -- 172.21.15.97:0/4179546157 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1781421b0 con 0x7ff15c00eca0 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 1 -- 172.21.15.97:0/4179546157 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780aac50 con 0x7ff15c005b40 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff16f7fe640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.565+0000 7ff17e451640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 10 monclient(hunting): get_auth_request con 0x7ff15c005b40 auth_method 0 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 msgr2=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-07T07:32:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 msgr2=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.566+0000 7ff17e451640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.576+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.576+0000 7ff17d44f640 10 monclient(hunting): get_auth_request con 0x7ff15c00eca0 auth_method 0 2023-12-07T07:32:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.576+0000 7ff17d44f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.576+0000 7ff17d44f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.576+0000 7ff17d44f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.577+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 msgr2=0x7ff15c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-07T07:32:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.577+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 msgr2=0x7ff15c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.577+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.577+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.819+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.830+0000 7ff17e451640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.851+0000 7ff17d44f640 10 monclient(hunting): get_auth_request con 0x7ff15c00eca0 auth_method 0 2023-12-07T07:32:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.851+0000 7ff17d44f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.851+0000 7ff17d44f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.851+0000 7ff17d44f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:45.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.859+0000 7ff17e451640 10 monclient(hunting): get_auth_request con 0x7ff15c005b40 auth_method 0 2023-12-07T07:32:45.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.859+0000 7ff17e451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.859+0000 7ff17e451640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.859+0000 7ff17e451640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:45.867 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.865+0000 7ff17e451640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 msgr2=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-07T07:32:45.867 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.865+0000 7ff17e451640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 msgr2=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:45.867 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.865+0000 7ff17e451640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:45.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.865+0000 7ff17e451640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:45.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.968+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 msgr2=0x7ff15c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-07T07:32:45.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.968+0000 7ff17d44f640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 msgr2=0x7ff15c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-07T07:32:45.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.968+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:45.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:45.968+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-07T07:32:46.122 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.120+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484047811 0 0) 0x7ff15c0108b0 con 0x7ff15c009870 2023-12-07T07:32:46.122 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.120+0000 7ff17cc4e640 10 monclient(hunting): _negotiate 2023-12-07T07:32:46.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.120+0000 7ff17cc4e640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:46.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.120+0000 7ff17cc4e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:46.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.120+0000 7ff17cc4e640 10 monclient(hunting): my global_id is 4098 2023-12-07T07:32:46.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.120+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff164003520 con 0x7ff15c009870 2023-12-07T07:32:46.190 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735501813 0 0) 0x7ff164003520 con 0x7ff15c009870 2023-12-07T07:32:46.190 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 msgr2=0x7ff15c0086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff15c0086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 msgr2=0x7ff15c00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 --2- 172.21.15.97:0/4179546157 >> v2:172.21.15.154:3300/0 conn(0x7ff15c00eca0 0x7ff15c00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: _finish_hunting 0 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 monclient: found mon.noname-a-legacy 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.97:6789/0 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1780aa970 con 0x7ff15c009870 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: _finish_auth 0 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: _check_auth_tickets 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2176119509 0 0) 0x7ff174007660 con 0x7ff15c009870 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17feb4640 5 monclient: authenticate success, global_id 4098 2023-12-07T07:32:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17feb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: dump: 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:46.192 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] but i'm connected to v1:172.21.15.97:6789/0, reconnecting 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 >> v1:172.21.15.97:6789/0 conn(0x7ff15c009870 legacy=0x7ff15c00fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 0x7ff164003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient(hunting): picked mon.b con 0x7ff1640039a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff1640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient(hunting): picked mon.c con 0x7ff15c005b40 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff164004d00 0x7ff164005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient(hunting): picked mon.a con 0x7ff164004d00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.186+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1781421b0 con 0x7ff164004d00 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17cc4e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780aac50 con 0x7ff1640039a0 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17cc4e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16400bb60 con 0x7ff15c005b40 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17cc4e640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17cc4e640 10 monclient(hunting): _finish_auth 0 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17feb4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 0x7ff164003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17d44f640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff1640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17dc50640 10 monclient(hunting): get_auth_request con 0x7ff1640039a0 auth_method 0 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17dc50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17dc50640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.187+0000 7ff17dc50640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.188+0000 7ff17d44f640 10 monclient(hunting): get_auth_request con 0x7ff15c005b40 auth_method 0 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.188+0000 7ff17d44f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.188+0000 7ff17d44f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:46.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.188+0000 7ff17d44f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:46.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.189+0000 7ff17dc50640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:46.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.189+0000 7ff17dc50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:46.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.189+0000 7ff17dc50640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:46.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.193+0000 7ff17d44f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:46.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.193+0000 7ff17d44f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:46.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.193+0000 7ff17d44f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:46.231 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.228+0000 7ff17e451640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff164004d00 0x7ff164005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:46.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.230+0000 7ff17e451640 10 monclient(hunting): get_auth_request con 0x7ff164004d00 auth_method 0 2023-12-07T07:32:46.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.230+0000 7ff17e451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:46.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.230+0000 7ff17e451640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:46.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.230+0000 7ff17e451640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:46.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17e451640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:46.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17e451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:46.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17e451640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:46.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-12-07T07:32:46.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 msgr2=0x7ff1640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff15c005b40 0x7ff1640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff164004d00 msgr2=0x7ff164005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff164004d00 0x7ff164005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 10 monclient: _finish_hunting 0 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 1 monclient: found mon.b 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 1 -- 172.21.15.97:0/4179546157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff164003520 con 0x7ff1640039a0 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 10 monclient: _finish_auth 0 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 10 monclient: _check_auth_tickets 2023-12-07T07:32:46.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.233+0000 7ff17dc50640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:46.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.241+0000 7ff17dc50640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 0x7ff164003dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff174003cb0 tx=0x7ff1640057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fd08e11cafea644 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17402dd30 con 0x7ff1640039a0 2023-12-07T07:32:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 10 monclient: dump: 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17400f040 con 0x7ff1640039a0 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 1 -- 172.21.15.97:0/4179546157 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17400b310 con 0x7ff1640039a0 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17cc4e640 10 monclient: dump: 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17feb4640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.305+0000 7ff17feb4640 4 set_mon_vals no callback set 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 10 monclient: shutdown 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 1 -- 172.21.15.97:0/4179546157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 msgr2=0x7ff164003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 1 --2- 172.21.15.97:0/4179546157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 0x7ff164003dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff174003cb0 tx=0x7ff1640057f0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17e451640 1 -- 172.21.15.97:0/4179546157 reap_dead start 2023-12-07T07:32:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 1 -- 172.21.15.97:0/4179546157 shutdown_connections 2023-12-07T07:32:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 1 -- 172.21.15.97:0/4179546157 >> 172.21.15.97:0/4179546157 conn(0x7ff178061a90 msgr2=0x7ff1780b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 1 -- 172.21.15.97:0/4179546157 shutdown_connections 2023-12-07T07:32:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.306+0000 7ff17feb4640 1 -- 172.21.15.97:0/4179546157 wait complete. 2023-12-07T07:32:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 10 monclient: build_initial_monmap 2023-12-07T07:32:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 10 monclient: monmap: 2023-12-07T07:32:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:32:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:32:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 librados: starting msgr at 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 librados: starting objecter 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 Processor -- start 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 -- start start 2023-12-07T07:32:46.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 librados: setting wanted keys 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 librados: calling monclient init 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 10 monclient: init 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 0x7ff1781d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 10 monclient(hunting): picked mon.noname-b con 0x7ff1640039a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff1780ab8a0 0x7ff1781d7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 10 monclient(hunting): picked mon.noname-c con 0x7ff1780ab8a0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.307+0000 7ff17feb4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1781d7980 0x7ff1781da190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 10 monclient(hunting): picked mon.noname-a con 0x7ff1781d7980 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780ac9f0 con 0x7ff1781d7980 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780acc20 con 0x7ff1640039a0 2023-12-07T07:32:46.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:46.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780b7e00 con 0x7ff1780ab8a0 2023-12-07T07:32:46.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:46.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.308+0000 7ff17feb4640 10 monclient(hunting): authenticate will time out at 1288.686412s 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.313+0000 7ff17e451640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1781d7980 0x7ff1781da190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1781d7980 0x7ff1781da190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33300/0 (socket says 172.21.15.97:33300) 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 1 -- 172.21.15.97:0/1120415377 learned_addr learned my addr 172.21.15.97:0/1120415377 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 10 monclient(hunting): get_auth_request con 0x7ff1781d7980 auth_method 0 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:46.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.314+0000 7ff17e451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 10 monclient(hunting): handle_auth_done global_id 4107 payload 824 2023-12-07T07:32:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 1 -- 172.21.15.97:0/1120415377 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff1780ab8a0 msgr2=0x7ff1781d7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 1 --2- 172.21.15.97:0/1120415377 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff1780ab8a0 0x7ff1781d7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 1 -- 172.21.15.97:0/1120415377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 msgr2=0x7ff1781d6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 1 --2- 172.21.15.97:0/1120415377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 0x7ff1781d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 10 monclient: _finish_hunting 0 2023-12-07T07:32:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 1 monclient: found mon.noname-a 2023-12-07T07:32:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:32:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.325+0000 7ff17e451640 1 -- 172.21.15.97:0/1120415377 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1780b53f0 con 0x7ff1781d7980 2023-12-07T07:32:46.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.336+0000 7ff17e451640 1 --2- 172.21.15.97:0/1120415377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1781d7980 0x7ff1781da190 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0061a0 tx=0x7ff1700053e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:46.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.336+0000 7ff16e7fc640 10 client.?.objecter ms_handle_connect 0x7ff1781d7980 2023-12-07T07:32:46.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.336+0000 7ff16e7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 1 -- 172.21.15.97:0/1120415377 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170011050 con 0x7ff1781d7980 2023-12-07T07:32:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:32:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: dump: 2023-12-07T07:32:46.383 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: _finish_auth 0 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:32:46.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 1 -- 172.21.15.97:0/1120415377 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff170012040 con 0x7ff1781d7980 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 5 monclient: authenticate success, global_id 4107 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 10 monclient: _renew_subs 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1780b8090 con 0x7ff1781d7980 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 1 -- 172.21.15.97:0/1120415377 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170016600 con 0x7ff1781d7980 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16e7fc640 10 monclient: dump: 2023-12-07T07:32:46.385 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff16ffff640 4 set_mon_vals no callback set 2023-12-07T07:32:46.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 10 monclient: _renew_subs 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1781da8b0 con 0x7ff1781d7980 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.380+0000 7ff17feb4640 1 librados: init done 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.381+0000 7ff17feb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.381+0000 7ff17feb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.381+0000 7ff17feb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:46.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.381+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff178067ac0 con 0x7ff1781d7980 2023-12-07T07:32:46.571 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 1 -- 172.21.15.97:0/1120415377 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff170016c70 con 0x7ff1781d7980 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 1 -- 172.21.15.97:0/1120415377 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff17002acf0 con 0x7ff1781d7980 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 10 client.4107.objecter ms_dispatch 0x7ff178000b90 osd_map(1..1 src has 1..1) v4 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 20 client.4107.objecter dump_active .. 0 homeless 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 1 -- 172.21.15.97:0/1120415377 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff1700204c0 con 0x7ff1781d7980 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:46.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.569+0000 7ff16e7fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:46.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.635+0000 7ff17feb4640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-07T07:32:46.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.635+0000 7ff17feb4640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-07T07:32:46.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.635+0000 7ff17feb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:46.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.635+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff178075b60 con 0x7ff1781d7980 2023-12-07T07:32:46.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.731+0000 7ff16e7fc640 1 -- 172.21.15.97:0/1120415377 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff17002a490 con 0x7ff1781d7980 2023-12-07T07:32:46.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.731+0000 7ff16e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-07T07:32:46.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.731+0000 7ff16e7fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-12-07T07:32:46.734 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2023-12-07T07:32:46.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 10 librados: watch_flush enter 2023-12-07T07:32:46.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 10 librados: watch_flush exit 2023-12-07T07:32:46.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 20 client.4107.objecter shutdown clearing up homeless session... 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 10 client.4107.objecter successfully canceled tick 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 10 monclient: shutdown 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1781d7980 msgr2=0x7ff1781da190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 --2- 172.21.15.97:0/1120415377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1781d7980 0x7ff1781da190 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0061a0 tx=0x7ff1700053e0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 shutdown_connections 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 --2- 172.21.15.97:0/1120415377 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1781d7980 0x7ff1781da190 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 --2- 172.21.15.97:0/1120415377 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff1780ab8a0 0x7ff1781d7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 --2- 172.21.15.97:0/1120415377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1640039a0 0x7ff1781d6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 >> 172.21.15.97:0/1120415377 conn(0x7ff178061a90 msgr2=0x7ff178138f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 shutdown_connections 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 -- 172.21.15.97:0/1120415377 wait complete. 2023-12-07T07:32:46.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:46.732+0000 7ff17feb4640 1 librados: shutdown 2023-12-07T07:32:46.749 INFO:tasks.ceph:check_enable_crimson: False 2023-12-07T07:32:46.749 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-07T07:32:46.749 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:46.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-07T07:32:46.779 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:46.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-07T07:32:46.846 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:46.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-07T07:32:46.913 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:32:46.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-07T07:32:46.917 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:46.915+0000 7f141e62b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-07T07:32:46.943 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:46.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-07T07:32:46.944 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:46.942+0000 7f753c845040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-07T07:32:46.970 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:46.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-07T07:32:46.971 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:46.969+0000 7f141e62b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-07T07:32:46.998 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:46.996+0000 7f753c845040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-07T07:32:47.024 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:47.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-07T07:32:47.095 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-12-07T07:32:47.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-07T07:32:47.161 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 3bb9fa9f-1af4-46cd-af57-26660874af7a 0 2023-12-07T07:32:47.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.300+0000 7f1f59889640 10 monclient: get_monmap_and_config 2023-12-07T07:32:47.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.300+0000 7f1f59889640 10 monclient: build_initial_monmap 2023-12-07T07:32:47.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.300+0000 7f1f59889640 10 monclient: monmap: 2023-12-07T07:32:47.302 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:47.302 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.302 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:47.300923+0000 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:47.300923+0000 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.300+0000 7f1f59889640 1 Processor -- start 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.300+0000 7f1f59889640 1 -- start start 2023-12-07T07:32:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.300+0000 7f1f59889640 10 monclient: init 2023-12-07T07:32:47.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:47.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-12-07T07:32:47.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1f540a8a00 0x7f1f540a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient(hunting): picked mon.noname-c con 0x7f1f540a8a00 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:47.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f540b2260 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:47.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1f5413f080 0x7f1f5413f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient(hunting): picked mon.noname-a con 0x7f1f5413f080 addr v2:172.21.15.97:3300/0 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f540aa970 con 0x7f1f540b2260 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f541421b0 con 0x7f1f5413f080 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f540aac50 con 0x7f1f540a8a00 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f59889640 10 monclient(hunting): authenticate will time out at 1261.658461s 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.301+0000 7f1f52ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1f540a8a00 0x7f1f540a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f537fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1f5413f080 0x7f1f5413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f52ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1f540a8a00 0x7f1f540a8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:39104/0 (socket says 172.21.15.154:39104) 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f52ffd640 1 -- 172.21.15.154:0/3312775022 learned_addr learned my addr 172.21.15.154:0/3312775022 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f52ffd640 10 monclient(hunting): get_auth_request con 0x7f1f540a8a00 auth_method 0 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f52ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f537fe640 10 monclient(hunting): get_auth_request con 0x7f1f5413f080 auth_method 0 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f537fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.302+0000 7f1f537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.303+0000 7f1f537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.303+0000 7f1f537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.303+0000 7f1f537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 10 monclient(hunting): handle_auth_done global_id 4110 payload 274 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 1 -- 172.21.15.154:0/3312775022 >> v2:172.21.15.154:3300/0 conn(0x7f1f540a8a00 msgr2=0x7f1f540a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 1 --2- 172.21.15.154:0/3312775022 >> v2:172.21.15.154:3300/0 conn(0x7f1f540a8a00 0x7f1f540a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 1 -- 172.21.15.154:0/3312775022 >> v1:172.21.15.154:6789/0 conn(0x7f1f540b2260 legacy=0x7f1f541425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 10 monclient: _finish_hunting 0 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 1 monclient: found mon.noname-a 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.304+0000 7f1f537fe640 1 -- 172.21.15.154:0/3312775022 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540aaf30 con 0x7f1f5413f080 2023-12-07T07:32:47.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.306+0000 7f1f537fe640 1 --2- 172.21.15.154:0/3312775022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f5413f530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1f540b8870 tx=0x7f1f4c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482cdf6bd34de834 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 1 -- 172.21.15.154:0/3312775022 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c02cb60 con 0x7f1f5413f080 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: dump: 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.312 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: _finish_auth 0 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 5 monclient: authenticate success, global_id 4110 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 1 -- 172.21.15.154:0/3312775022 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c02ccc0 con 0x7f1f5413f080 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:47.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 4 set_mon_vals no callback set 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 1 -- 172.21.15.154:0/3312775022 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c034660 con 0x7f1f5413f080 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f51ffb640 10 monclient: dump: 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.314 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.315 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.315 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.315 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.315 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.315 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 10 monclient: shutdown 2023-12-07T07:32:47.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:47.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 -- 172.21.15.154:0/3312775022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 msgr2=0x7f1f5413f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 --2- 172.21.15.154:0/3312775022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f5413f530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1f540b8870 tx=0x7f1f4c02bdb0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 -- 172.21.15.154:0/3312775022 shutdown_connections 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 --2- 172.21.15.154:0/3312775022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f5413f530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 --2- 172.21.15.154:0/3312775022 >> v2:172.21.15.154:3300/0 conn(0x7f1f540a8a00 0x7f1f540a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 -- 172.21.15.154:0/3312775022 >> 172.21.15.154:0/3312775022 conn(0x7f1f54061a90 msgr2=0x7f1f540b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 -- 172.21.15.154:0/3312775022 shutdown_connections 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.310+0000 7f1f59889640 1 -- 172.21.15.154:0/3312775022 wait complete. 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.311+0000 7f1f59889640 10 monclient: build_initial_monmap 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.311+0000 7f1f59889640 10 monclient: monmap: 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:47.319 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.311+0000 7f1f59889640 1 librados: starting msgr at 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.311+0000 7f1f59889640 1 librados: starting objecter 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.311+0000 7f1f59889640 1 Processor -- start 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 -- start start 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 librados: setting wanted keys 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 librados: calling monclient init 2023-12-07T07:32:47.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient: init 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 0x7f1f5413bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): picked mon.noname-b con 0x7f1f540a8a00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f54143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): picked mon.noname-c con 0x7f1f540b2260 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f54144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): picked mon.noname-a con 0x7f1f5413f080 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f541454c0 con 0x7f1f5413f080 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f541367e0 con 0x7f1f540a8a00 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5413c590 con 0x7f1f540b2260 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.312+0000 7f1f59889640 10 monclient(hunting): authenticate will time out at 1261.669967s 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f52ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 0x7f1f5413bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f54144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f527fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f54143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f52ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 0x7f1f5413bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:39108/0 (socket says 172.21.15.154:39108) 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 learned_addr learned my addr 172.21.15.154:0/1698024817 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f52ffd640 10 monclient(hunting): get_auth_request con 0x7f1f540a8a00 auth_method 0 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f52ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f537fe640 10 monclient(hunting): get_auth_request con 0x7f1f5413f080 auth_method 0 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f537fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f527fc640 10 monclient(hunting): get_auth_request con 0x7f1f540b2260 auth_method 0 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f527fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.313+0000 7f1f527fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.314+0000 7f1f527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 msgr2=0x7f1f54143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f54143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 msgr2=0x7f1f54144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f54144bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 10 monclient: _finish_hunting 0 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 1 monclient: found mon.noname-b 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:32:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.316+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c009a70 con 0x7f1f540a8a00 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.319+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 0x7f1f5413bb60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800e7e0 tx=0x7f1f4800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.319+0000 7f1f3bfff640 10 client.?.objecter ms_handle_connect 0x7f1f540a8a00 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.319+0000 7f1f3bfff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4800c8d0 con 0x7f1f540a8a00 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: dump: 2023-12-07T07:32:47.329 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: _finish_auth 0 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: _check_auth_tickets 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 5 monclient: authenticate success, global_id 4111 2023-12-07T07:32:47.330 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4800ee90 con 0x7f1f540a8a00 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4800cc70 con 0x7f1f540a8a00 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f3bfff640 10 monclient: dump: 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 10 monclient: _renew_subs 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f517fa640 4 set_mon_vals no callback set 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f541450f0 con 0x7f1f540a8a00 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 10 monclient: _renew_subs 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f54145b30 con 0x7f1f540a8a00 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.322+0000 7f1f59889640 1 librados: init done 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f59889640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f59889640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f59889640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f54067ac0 con 0x7f1f540a8a00 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 msgr2=0x7f1f5413bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-12-07T07:32:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 msgr2=0x7f1f5413bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 0x7f1f5413bb60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800e7e0 tx=0x7f1f4800eca0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 0x7f1f5413bb60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800e7e0 tx=0x7f1f4800eca0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 msgr2=0x7f1f5413bb60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f540a8a00 0x7f1f5413bb60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f40007250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 10 monclient(hunting): picked mon.c con 0x7f1f540b2260 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f40009c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 10 monclient(hunting): picked mon.a con 0x7f1f5413f080 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f4000a660 0x7f1f4000caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 10 monclient(hunting): picked mon.b con 0x7f1f4000a660 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.323+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4000d5a0 con 0x7f1f5413f080 2023-12-07T07:32:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f3bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4000d710 con 0x7f1f4000a660 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f3bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4000d880 con 0x7f1f540b2260 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f3bfff640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f40007250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f527fc640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f40009c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f52ffd640 10 monclient(hunting): get_auth_request con 0x7f1f540b2260 auth_method 0 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f52ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f52ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f527fc640 10 monclient(hunting): get_auth_request con 0x7f1f5413f080 auth_method 0 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f527fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.324+0000 7f1f527fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.325+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.325+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.325+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.325+0000 7f1f527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.325+0000 7f1f527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.325+0000 7f1f527fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 10 monclient(hunting): handle_auth_done global_id 4111 payload 293 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f4000a660 msgr2=0x7f1f4000caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f4000a660 0x7f1f4000caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 msgr2=0x7f1f40009c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f40009c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 10 monclient: _finish_hunting 0 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 1 monclient: found mon.c 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:47.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f1f4000db70 con 0x7f1f540b2260 2023-12-07T07:32:47.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:47.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f54067ac0 con 0x7f1f540b2260 2023-12-07T07:32:47.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 10 monclient: _finish_auth 0 2023-12-07T07:32:47.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:47.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.327+0000 7f1f52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:47.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.329+0000 7f1f52ffd640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f40007250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f480265c0 tx=0x7f1f48008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:47.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.329+0000 7f1f3bfff640 10 client.4111.objecter ms_handle_connect 0x7f1f540b2260 2023-12-07T07:32:47.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.329+0000 7f1f3bfff640 10 client.4111.objecter resend_mon_ops 2023-12-07T07:32:47.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.501+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f480267f0 con 0x7f1f540b2260 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.501+0000 7f1f3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.501+0000 7f1f3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.501+0000 7f1f3bfff640 10 monclient: dump: 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.501+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f48026990 con 0x7f1f540b2260 2023-12-07T07:32:47.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.501+0000 7f1f3bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:47.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.501+0000 7f1f50ff9640 4 set_mon_vals no callback set 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1f48031470 con 0x7f1f540b2260 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.2 v2:172.21.15.97:3301/0 4 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1f48026e40 con 0x7f1f540b2260 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 10 client.4111.objecter ms_dispatch 0x7f1f54000b90 osd_map(2..2 src has 1..2) v4 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 20 client.4111.objecter dump_active .. 0 homeless 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.2 v2:172.21.15.97:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1f48021060 con 0x7f1f540b2260 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:47.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.508+0000 7f1f3bfff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:47.538 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.536+0000 7f141e62b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-07T07:32:47.557 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.555+0000 7f753c845040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-07T07:32:47.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.580+0000 7f1f59889640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3bb9fa9f-1af4-46cd-af57-26660874af7a", "id": 0}] 2023-12-07T07:32:47.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.580+0000 7f1f59889640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3bb9fa9f-1af4-46cd-af57-26660874af7a", "id": 0}] 2023-12-07T07:32:47.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.580+0000 7f1f59889640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.580+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3bb9fa9f-1af4-46cd-af57-26660874af7a", "id": 0} v 0) v1 -- 0x7f1f54075b60 con 0x7f1f540b2260 2023-12-07T07:32:47.590 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.588+0000 7f141e62b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-07T07:32:47.610 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.608+0000 7f753c845040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-07T07:32:47.638 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.636+0000 7f141e62b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-07T07:32:47.655 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.653+0000 7f753c845040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-07T07:32:47.692 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.690+0000 7f141e62b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-07T07:32:47.708 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.707+0000 7f753c845040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-07T07:32:47.738 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.736+0000 7f141e62b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-07T07:32:47.752 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.750+0000 7f753c845040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-07T07:32:47.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.767+0000 7f1f3bfff640 1 -- 172.21.15.154:0/1698024817 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3bb9fa9f-1af4-46cd-af57-26660874af7a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1f48017070 con 0x7f1f540b2260 2023-12-07T07:32:47.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.767+0000 7f1f3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3bb9fa9f-1af4-46cd-af57-26660874af7a", "id": 0}] 2023-12-07T07:32:47.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.767+0000 7f1f3bfff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:47.770 INFO:teuthology.orchestra.run.smithi154.stdout:0 2023-12-07T07:32:47.771 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 10 librados: watch_flush enter 2023-12-07T07:32:47.771 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 10 librados: watch_flush exit 2023-12-07T07:32:47.771 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 20 client.4111.objecter shutdown clearing up homeless session... 2023-12-07T07:32:47.771 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 10 client.4111.objecter successfully canceled tick 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 10 monclient: shutdown 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 msgr2=0x7f1f40007250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f40007250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f480265c0 tx=0x7f1f48008040 comp rx=0 tx=0).stop 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 shutdown_connections 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f4000a660 0x7f1f4000caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f5413f080 0x7f1f40009c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 --2- 172.21.15.154:0/1698024817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1f540b2260 0x7f1f40007250 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 >> 172.21.15.154:0/1698024817 conn(0x7f1f54061a90 msgr2=0x7f1f5413e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 shutdown_connections 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.768+0000 7f1f59889640 1 -- 172.21.15.154:0/1698024817 wait complete. 2023-12-07T07:32:47.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.769+0000 7f1f59889640 1 librados: shutdown 2023-12-07T07:32:47.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb 1 2023-12-07T07:32:47.843 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.841+0000 7f141e62b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-07T07:32:47.859 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.857+0000 7f753c845040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-07T07:32:47.879 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.877+0000 7fdf11ae7640 10 monclient: get_monmap_and_config 2023-12-07T07:32:47.879 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.877+0000 7fdf11ae7640 10 monclient: build_initial_monmap 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.877+0000 7fdf11ae7640 10 monclient: monmap: 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:47.878477+0000 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:47.878477+0000 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:47.880 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 1 Processor -- start 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 1 -- start start 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 10 monclient: init 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdf0c0a8a00 0x7fdf0c0a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 10 monclient(hunting): picked mon.noname-a con 0x7fdf0c0a8a00 addr v2:172.21.15.97:3300/0 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c1425d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.878+0000 7fdf11ae7640 10 monclient(hunting): picked mon.noname-b con 0x7fdf0c0b2260 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:47.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf0c142b10 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0c0aa970 con 0x7fdf0c142b10 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf0c1421b0 con 0x7fdf0c0a8a00 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0aac50 con 0x7fdf0c0b2260 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf11ae7640 10 monclient(hunting): authenticate will time out at 1262.236238s 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf0affd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c1425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf0bfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fdf0c142b10 0x7fdf0c13f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:51204/0 (socket says 172.21.15.154:51204) 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.879+0000 7fdf0bfff640 1 -- 172.21.15.154:0/2538379901 learned_addr learned my addr 172.21.15.154:0/2538379901 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.880+0000 7fdf0affd640 10 monclient(hunting): get_auth_request con 0x7fdf0c0b2260 auth_method 0 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.880+0000 7fdf0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.880+0000 7fdf0affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.880+0000 7fdf0affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.880+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.880+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.880+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:47.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-12-07T07:32:47.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 1 -- 172.21.15.154:0/2538379901 >> v2:172.21.15.97:3300/0 conn(0x7fdf0c0a8a00 msgr2=0x7fdf0c0a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:47.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 1 --2- 172.21.15.154:0/2538379901 >> v2:172.21.15.97:3300/0 conn(0x7fdf0c0a8a00 0x7fdf0c0a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 1 -- 172.21.15.154:0/2538379901 >> v1:172.21.15.154:6789/0 conn(0x7fdf0c142b10 legacy=0x7fdf0c13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 10 monclient: _finish_hunting 0 2023-12-07T07:32:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 1 monclient: found mon.noname-b 2023-12-07T07:32:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.882+0000 7fdf0affd640 1 -- 172.21.15.154:0/2538379901 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c0aaf30 con 0x7fdf0c0b2260 2023-12-07T07:32:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.884+0000 7fdf0affd640 1 --2- 172.21.15.154:0/2538379901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c1425d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0097a0 tx=0x7fdefc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60580c1267a25e45 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 1 -- 172.21.15.154:0/2538379901 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc02f050 con 0x7fdf0c0b2260 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: dump: 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.890 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: _finish_auth 0 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 5 monclient: authenticate success, global_id 4112 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 1 -- 172.21.15.154:0/2538379901 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc031700 con 0x7fdf0c0b2260 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 4 set_mon_vals no callback set 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 1 -- 172.21.15.154:0/2538379901 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc031a90 con 0x7fdf0c0b2260 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf0a7fc640 10 monclient: dump: 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.892 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 10 monclient: shutdown 2023-12-07T07:32:47.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:47.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 -- 172.21.15.154:0/2538379901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 msgr2=0x7fdf0c1425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 --2- 172.21.15.154:0/2538379901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c1425d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0097a0 tx=0x7fdefc02b7a0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 -- 172.21.15.154:0/2538379901 shutdown_connections 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 --2- 172.21.15.154:0/2538379901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c1425d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 --2- 172.21.15.154:0/2538379901 >> v2:172.21.15.97:3300/0 conn(0x7fdf0c0a8a00 0x7fdf0c0a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 -- 172.21.15.154:0/2538379901 >> 172.21.15.154:0/2538379901 conn(0x7fdf0c061a90 msgr2=0x7fdf0c0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 -- 172.21.15.154:0/2538379901 shutdown_connections 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.888+0000 7fdf11ae7640 1 -- 172.21.15.154:0/2538379901 wait complete. 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.889+0000 7fdf11ae7640 10 monclient: build_initial_monmap 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.889+0000 7fdf11ae7640 10 monclient: monmap: 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.896 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.889+0000 7fdf11ae7640 1 librados: starting msgr at 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.889+0000 7fdf11ae7640 1 librados: starting objecter 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.889+0000 7fdf11ae7640 1 Processor -- start 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 -- start start 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 librados: setting wanted keys 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 librados: calling monclient init 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient: init 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdf0c0b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): picked mon.noname-a con 0x7fdf0c0a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): picked mon.noname-c con 0x7fdf0c0b2260 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:47.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdf0c143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:47.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): picked mon.noname-b con 0x7fdf0c142b10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:47.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0b64b0 con 0x7fdf0c0a8a00 2023-12-07T07:32:47.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0b6720 con 0x7fdf0c142b10 2023-12-07T07:32:47.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0b6a00 con 0x7fdf0c0b2260 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.890+0000 7fdf11ae7640 10 monclient(hunting): authenticate will time out at 1262.247891s 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0affd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdf0c143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0affd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c143300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.154:36258/0 (socket says 172.21.15.154:36258) 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 learned_addr learned my addr 172.21.15.154:0/1367609934 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0affd640 10 monclient(hunting): get_auth_request con 0x7fdf0c0b2260 auth_method 0 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0bfff640 10 monclient(hunting): get_auth_request con 0x7fdf0c142b10 auth_method 0 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.891+0000 7fdf0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:47.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.892+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.892+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.892+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:47.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.892+0000 7fdf0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:47.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.892+0000 7fdf0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:47.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.892+0000 7fdf0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 msgr2=0x7fdf0c143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdf0c143840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 msgr2=0x7fdf0c0b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdf0c0b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 10 monclient: _finish_hunting 0 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 1 monclient: found mon.noname-c 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.894+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc009400 con 0x7fdf0c0b2260 2023-12-07T07:32:47.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.897+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c143300 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0098c0 tx=0x7fdefc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:47.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.897+0000 7fdef3fff640 10 client.?.objecter ms_handle_connect 0x7fdf0c0b2260 2023-12-07T07:32:47.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.897+0000 7fdef3fff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc033c30 con 0x7fdf0c0b2260 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: dump: 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: _finish_auth 0 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: _check_auth_tickets 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc033d90 con 0x7fdf0c0b2260 2023-12-07T07:32:47.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc03c640 con 0x7fdf0c0b2260 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdef3fff640 10 monclient: dump: 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:47.908 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 5 monclient: authenticate success, global_id 4115 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 10 monclient: _renew_subs 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c144c00 con 0x7fdf0c0b2260 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf09ffb640 4 set_mon_vals no callback set 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 10 monclient: _renew_subs 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c1335c0 con 0x7fdf0c0b2260 2023-12-07T07:32:47.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.901+0000 7fdf11ae7640 1 librados: init done 2023-12-07T07:32:47.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.902+0000 7fdf11ae7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:47.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.902+0000 7fdf11ae7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:47.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.902+0000 7fdf11ae7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:47.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.902+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c067ac0 con 0x7fdf0c0b2260 2023-12-07T07:32:47.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.906+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdefc03a040 con 0x7fdf0c0b2260 2023-12-07T07:32:47.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.906+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fdefc052720 con 0x7fdf0c0b2260 2023-12-07T07:32:47.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.906+0000 7fdef3fff640 10 client.4115.objecter ms_dispatch 0x7fdf0c000b90 osd_map(3..3 src has 1..3) v4 2023-12-07T07:32:47.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.906+0000 7fdef3fff640 3 client.4115.objecter handle_osd_map got epochs [3,3] > 0 2023-12-07T07:32:47.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.906+0000 7fdef3fff640 3 client.4115.objecter handle_osd_map decoding full epoch 3 2023-12-07T07:32:47.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:47.906+0000 7fdef3fff640 20 client.4115.objecter dump_active .. 0 homeless 2023-12-07T07:32:47.921 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.919+0000 7f141e62b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-07T07:32:47.934 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.932+0000 7f753c845040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-07T07:32:47.970 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:47.968+0000 7f141e62b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-07T07:32:47.980 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:47.978+0000 7f753c845040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-07T07:32:48.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.008+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fdefc043020 con 0x7fdf0c0b2260 2023-12-07T07:32:48.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.008+0000 7fdef3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:48.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.008+0000 7fdef3fff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:48.024 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:48.022+0000 7f141e62b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-07T07:32:48.033 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:48.031+0000 7f753c845040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-07T07:32:48.080 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:48.078+0000 7f141e62b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-07T07:32:48.082 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.080+0000 7fdf11ae7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1}] 2023-12-07T07:32:48.082 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.080+0000 7fdf11ae7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1}] 2023-12-07T07:32:48.082 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.080+0000 7fdf11ae7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.082 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.080+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1} v 0) v1 -- 0x7fdf0c0b6f20 con 0x7fdf0c0b2260 2023-12-07T07:32:48.084 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:48.082+0000 7f753c845040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-07T07:32:48.087 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 msgr2=0x7fdf0c143300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-12-07T07:32:48.087 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 msgr2=0x7fdf0c143300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-07T07:32:48.087 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c143300 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0098c0 tx=0x7fdefc0046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c143300 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0098c0 tx=0x7fdefc0046c0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 msgr2=0x7fdf0c143300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdf0c0b2260 0x7fdf0c143300 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient(hunting): picked mon.c con 0x7fdef8006f20 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdef800bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient(hunting): picked mon.b con 0x7fdf0c142b10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdef800e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient(hunting): picked mon.a con 0x7fdf0c0a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef800f110 con 0x7fdf0c0a8a00 2023-12-07T07:32:48.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef800f250 con 0x7fdf0c142b10 2023-12-07T07:32:48.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef800f3e0 con 0x7fdef8006f20 2023-12-07T07:32:48.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.085+0000 7fdef3fff640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:48.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.091+0000 7fdf0b7fe640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.091+0000 7fdf0b7fe640 10 monclient(hunting): get_auth_request con 0x7fdef8006f20 auth_method 0 2023-12-07T07:32:48.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.091+0000 7fdf0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.091+0000 7fdf0b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.091+0000 7fdf0b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:48.100 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.098+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdef800bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.100+0000 7fdf0affd640 10 monclient(hunting): get_auth_request con 0x7fdf0c142b10 auth_method 0 2023-12-07T07:32:48.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.100+0000 7fdf0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.100+0000 7fdf0affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.100+0000 7fdf0affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:48.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.100+0000 7fdf0bfff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdef800e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.103 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.101+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.103 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.101+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.103 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.101+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.102+0000 7fdf0bfff640 10 monclient(hunting): get_auth_request con 0x7fdf0c0a8a00 auth_method 0 2023-12-07T07:32:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.102+0000 7fdf0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.102+0000 7fdf0bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.102+0000 7fdf0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.102+0000 7fdf0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.102+0000 7fdf0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.102+0000 7fdf0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:48.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 10 monclient(hunting): handle_auth_done global_id 4115 payload 293 2023-12-07T07:32:48.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 msgr2=0x7fdef8009330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 msgr2=0x7fdef800e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdef800e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 10 monclient: _finish_hunting 0 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 1 monclient: found mon.b 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fdef800f570 con 0x7fdf0c142b10 2023-12-07T07:32:48.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1}] 2023-12-07T07:32:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1} v 0) v1 -- 0x7fdf0c067ac0 con 0x7fdf0c142b10 2023-12-07T07:32:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 10 monclient: _finish_auth 0 2023-12-07T07:32:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.103+0000 7fdf0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:48.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.105+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdef800bc30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0526f0 tx=0x7fdefc005d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:48.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.105+0000 7fdef3fff640 10 client.4115.objecter ms_handle_connect 0x7fdf0c142b10 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.105+0000 7fdef3fff640 10 client.4115.objecter resend_mon_ops 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.108+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc02f020 con 0x7fdf0c142b10 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.108+0000 7fdef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.108+0000 7fdef3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.108+0000 7fdef3fff640 10 monclient: dump: 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.111 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.150 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.147+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 msgr2=0x7fdef800bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.147+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 msgr2=0x7fdef800bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.147+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdef800bc30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0526f0 tx=0x7fdefc005d10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdef800bc30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0526f0 tx=0x7fdefc005d10 comp rx=0 tx=0).stop 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 msgr2=0x7fdef800bc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c142b10 0x7fdef800bc30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdef800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient(hunting): picked mon.a con 0x7fdf0c0a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient(hunting): picked mon.c con 0x7fdef8006f20 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c0b2260 0x7fdef8011d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient(hunting): picked mon.b con 0x7fdf0c0b2260 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8012760 con 0x7fdf0c0a8a00 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80128a0 con 0x7fdf0c0b2260 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80129e0 con 0x7fdef8006f20 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdef3fff640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.148+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.149+0000 7fdf0affd640 10 monclient(hunting): get_auth_request con 0x7fdef8006f20 auth_method 0 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.149+0000 7fdf0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.149+0000 7fdf0affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.149+0000 7fdf0affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.149+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.149+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.149+0000 7fdf0affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 10 monclient(hunting): handle_auth_done global_id 4115 payload 293 2023-12-07T07:32:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c0b2260 msgr2=0x7fdef8011d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c0b2260 0x7fdef8011d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 msgr2=0x7fdef800e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdef800e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 10 monclient: _finish_hunting 0 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 1 monclient: found mon.c 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fdef8012b50 con 0x7fdef8006f20 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1}] 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 1 -- 172.21.15.154:0/1367609934 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1} v 0) v1 -- 0x7fdf0c067ac0 con 0x7fdef8006f20 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 10 monclient: _finish_auth 0 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:48.154 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.151+0000 7fdf0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:48.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.153+0000 7fdef3fff640 10 client.4115.objecter ms_handle_connect 0x7fdef8006f20 2023-12-07T07:32:48.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.153+0000 7fdf0affd640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdefc02bdb0 tx=0x7fdefc052930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:48.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.153+0000 7fdef3fff640 10 client.4115.objecter resend_mon_ops 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc052ca0 con 0x7fdef8006f20 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdef3fff640 10 monclient: dump: 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.166 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc052e00 con 0x7fdef8006f20 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdef3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdefc0620b0 con 0x7fdef8006f20 2023-12-07T07:32:48.167 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.164+0000 7fdf09ffb640 4 set_mon_vals no callback set 2023-12-07T07:32:48.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.276+0000 7fdef3fff640 1 -- 172.21.15.154:0/1367609934 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdf0c0b6f20 con 0x7fdef8006f20 2023-12-07T07:32:48.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.276+0000 7fdef3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb", "id": 1}] 2023-12-07T07:32:48.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.276+0000 7fdef3fff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:48.279 INFO:teuthology.orchestra.run.smithi154.stdout:1 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 10 librados: watch_flush enter 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 10 librados: watch_flush exit 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 20 client.4115.objecter shutdown clearing up homeless session... 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 10 client.4115.objecter successfully canceled tick 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 10 monclient: shutdown 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 msgr2=0x7fdef8009330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdefc02bdb0 tx=0x7fdefc052930 comp rx=0 tx=0).stop 2023-12-07T07:32:48.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 shutdown_connections 2023-12-07T07:32:48.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf0c0b2260 0x7fdef8011d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdef8006f20 0x7fdef8009330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 --2- 172.21.15.154:0/1367609934 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c0a8a00 0x7fdef800e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 >> 172.21.15.154:0/1367609934 conn(0x7fdf0c061a90 msgr2=0x7fdf0c13e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:48.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 shutdown_connections 2023-12-07T07:32:48.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 -- 172.21.15.154:0/1367609934 wait complete. 2023-12-07T07:32:48.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.277+0000 7fdf11ae7640 1 librados: shutdown 2023-12-07T07:32:48.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 2e035cab-ae59-4d53-b46e-867d865bfc46 2 2023-12-07T07:32:48.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.387+0000 7fb620b5a640 10 monclient: get_monmap_and_config 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.387+0000 7fb620b5a640 10 monclient: build_initial_monmap 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.387+0000 7fb620b5a640 10 monclient: monmap: 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:48.388422+0000 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:48.388422+0000 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 1 Processor -- start 2023-12-07T07:32:48.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 1 -- start start 2023-12-07T07:32:48.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient: init 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb61c0a8a00 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb61c0b2260 0x7fb61c0a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c0b2260 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c13f080 0x7fb61c13f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c13f080 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c0aa970 con 0x7fb61c0a8a00 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61c1421b0 con 0x7fb61c0b2260 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c0aac50 con 0x7fb61c13f080 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.388+0000 7fb620b5a640 10 monclient(hunting): authenticate will time out at 1262.746108s 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb619d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb61c0b2260 0x7fb61c0a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb61ad9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c13f080 0x7fb61c13f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb61ad9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c13f080 0x7fb61c13f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.154:36276/0 (socket says 172.21.15.154:36276) 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb61ad9d640 1 -- 172.21.15.154:0/2475693439 learned_addr learned my addr 172.21.15.154:0/2475693439 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb61ad9d640 10 monclient(hunting): get_auth_request con 0x7fb61c13f080 auth_method 0 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb61ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb61ad9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.389+0000 7fb61ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb619d9b640 10 monclient(hunting): get_auth_request con 0x7fb61c0b2260 auth_method 0 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb619d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb619d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb619d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.390+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.391+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.391+0000 7fb61ad9d640 1 -- 172.21.15.154:0/2475693439 >> v2:172.21.15.154:3300/0 conn(0x7fb61c0b2260 msgr2=0x7fb61c0a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.391+0000 7fb61ad9d640 1 --2- 172.21.15.154:0/2475693439 >> v2:172.21.15.154:3300/0 conn(0x7fb61c0b2260 0x7fb61c0a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.392+0000 7fb61ad9d640 1 -- 172.21.15.154:0/2475693439 >> v1:172.21.15.154:6789/0 conn(0x7fb61c0a8a00 legacy=0x7fb61c1425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.392+0000 7fb61ad9d640 10 monclient: _finish_hunting 0 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.392+0000 7fb61ad9d640 1 monclient: found mon.noname-b 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.392+0000 7fb61ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.392+0000 7fb61ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.392+0000 7fb61ad9d640 1 -- 172.21.15.154:0/2475693439 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c0aaf30 con 0x7fb61c13f080 2023-12-07T07:32:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.393+0000 7fb61ad9d640 1 --2- 172.21.15.154:0/2475693439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c13f080 0x7fb61c13f530 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb614009e50 tx=0x7fb61402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1af31b0c311d9469 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:48.403 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 1 -- 172.21.15.154:0/2475693439 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61402c9e0 con 0x7fb61c13f080 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: dump: 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: _finish_auth 0 2023-12-07T07:32:48.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: _check_auth_tickets 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 1 -- 172.21.15.154:0/2475693439 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61402cb40 con 0x7fb61c13f080 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 1 -- 172.21.15.154:0/2475693439 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6140343f0 con 0x7fb61c13f080 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb620b5a640 5 monclient: authenticate success, global_id 4118 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb620b5a640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb620b5a640 4 set_mon_vals no callback set 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.397+0000 7fb61959a640 10 monclient: dump: 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.405 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 10 monclient: shutdown 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 -- 172.21.15.154:0/2475693439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c13f080 msgr2=0x7fb61c13f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 --2- 172.21.15.154:0/2475693439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c13f080 0x7fb61c13f530 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb614009e50 tx=0x7fb61402be40 comp rx=0 tx=0).stop 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 -- 172.21.15.154:0/2475693439 shutdown_connections 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 --2- 172.21.15.154:0/2475693439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c13f080 0x7fb61c13f530 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 --2- 172.21.15.154:0/2475693439 >> v2:172.21.15.154:3300/0 conn(0x7fb61c0b2260 0x7fb61c0a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 -- 172.21.15.154:0/2475693439 >> 172.21.15.154:0/2475693439 conn(0x7fb61c061a90 msgr2=0x7fb61c0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:48.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 -- 172.21.15.154:0/2475693439 shutdown_connections 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.398+0000 7fb620b5a640 1 -- 172.21.15.154:0/2475693439 wait complete. 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient: build_initial_monmap 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient: monmap: 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 librados: starting msgr at 2023-12-07T07:32:48.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 librados: starting objecter 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 Processor -- start 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 -- start start 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 librados: setting wanted keys 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 librados: calling monclient init 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient: init 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb61c143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c0a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c0b2260 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb61c145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.399+0000 7fb620b5a640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c13f080 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c144fc0 con 0x7fb61c0a8a00 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c1435a0 con 0x7fb61c13f080 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c1451f0 con 0x7fb61c0b2260 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb620b5a640 10 monclient(hunting): authenticate will time out at 1262.757218s 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb61ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb61c145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb619d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb61a59c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb61c143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb619d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.154:36286/0 (socket says 172.21.15.154:36286) 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb619d9b640 1 -- 172.21.15.154:0/236069033 learned_addr learned my addr 172.21.15.154:0/236069033 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb61ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb61c145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:39160/0 (socket says 172.21.15.154:39160) 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb619d9b640 10 monclient(hunting): get_auth_request con 0x7fb61c0b2260 auth_method 0 2023-12-07T07:32:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb619d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb619d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb619d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb61ad9d640 10 monclient(hunting): get_auth_request con 0x7fb61c13f080 auth_method 0 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb61ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb61ad9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.400+0000 7fb61ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61a59c640 10 monclient(hunting): get_auth_request con 0x7fb61c0a8a00 auth_method 0 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61a59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61a59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.401+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 10 monclient(hunting): handle_auth_done global_id 4121 payload 824 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 msgr2=0x7fb61c145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb61c145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 msgr2=0x7fb61c143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb61c143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 10 monclient: _finish_hunting 0 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 1 monclient: found mon.noname-c 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.404+0000 7fb619d9b640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb614009b30 con 0x7fb61c0b2260 2023-12-07T07:32:48.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.405+0000 7fb619d9b640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 msgr2=0x7fb61c13c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-12-07T07:32:48.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.405+0000 7fb619d9b640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 msgr2=0x7fb61c13c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-07T07:32:48.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.405+0000 7fb619d9b640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fb61000e980 tx=0x7fb61000ee40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-07T07:32:48.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.405+0000 7fb619d9b640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-07T07:32:48.488 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:48.486+0000 7f753c845040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-07T07:32:48.496 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:48.494+0000 7f141e62b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-07T07:32:48.610 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.606+0000 7fb619d9b640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb619d9b640 10 monclient: get_auth_request con 0x7fb61c0b2260 auth_method 0 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb619d9b640 0 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb619d9b640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 msgr2=0x7fb61c13c350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb61c0b2260 0x7fb61c13c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb5fc0054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c13f080 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5fc005f80 0x7fb5fc008410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient(hunting): picked mon.noname-c con 0x7fb5fc005f80 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb5fc00afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c0a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc00b9e0 con 0x7fb61c0a8a00 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc00bb50 con 0x7fb61c13f080 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc00bd10 con 0x7fb5fc005f80 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb60affd640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb61a59c640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb5fc0054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.607+0000 7fb61ad9d640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb5fc00afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61a59c640 10 monclient(hunting): get_auth_request con 0x7fb61c13f080 auth_method 0 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61a59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:48.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61ad9d640 10 monclient(hunting): get_auth_request con 0x7fb61c0a8a00 auth_method 0 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61ad9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb619d9b640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5fc005f80 0x7fb5fc008410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb61a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb619d9b640 10 monclient(hunting): get_auth_request con 0x7fb5fc005f80 auth_method 0 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb619d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb619d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.608+0000 7fb619d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:32:48.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.609+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.609+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.609+0000 7fb619d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:32:48.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 10 monclient(hunting): handle_auth_done global_id 4121 payload 293 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5fc005f80 msgr2=0x7fb5fc008410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5fc005f80 0x7fb5fc008410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 msgr2=0x7fb5fc0054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb5fc0054d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 10 monclient: _finish_hunting 0 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 1 monclient: found mon.noname-a 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.610+0000 7fb61ad9d640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc00bf30 con 0x7fb61c0a8a00 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.612+0000 7fb61ad9d640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb5fc00afd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb614032000 tx=0x7fb610009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.612+0000 7fb60affd640 10 client.?.objecter ms_handle_connect 0x7fb61c0a8a00 2023-12-07T07:32:48.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.612+0000 7fb60affd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614047610 con 0x7fb61c0a8a00 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: dump: 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: _finish_auth 0 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb614047770 con 0x7fb61c0a8a00 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614047b20 con 0x7fb61c0a8a00 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb60affd640 10 monclient: dump: 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 5 monclient: authenticate success, global_id 4121 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 10 monclient: _renew_subs 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb618d99640 4 set_mon_vals no callback set 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c13b510 con 0x7fb61c0a8a00 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 10 monclient: _renew_subs 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c13c350 con 0x7fb61c0a8a00 2023-12-07T07:32:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.616+0000 7fb620b5a640 1 librados: init done 2023-12-07T07:32:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.617+0000 7fb620b5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.617+0000 7fb620b5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.617+0000 7fb620b5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.617+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61c067ac0 con 0x7fb61c0a8a00 2023-12-07T07:32:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.619+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb614003990 con 0x7fb61c0a8a00 2023-12-07T07:32:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.619+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fb61403f410 con 0x7fb61c0a8a00 2023-12-07T07:32:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.619+0000 7fb60affd640 10 client.4121.objecter ms_dispatch 0x7fb61c000b90 osd_map(4..4 src has 1..4) v4 2023-12-07T07:32:48.623 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.619+0000 7fb60affd640 3 client.4121.objecter handle_osd_map got epochs [4,4] > 0 2023-12-07T07:32:48.623 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.619+0000 7fb60affd640 3 client.4121.objecter handle_osd_map decoding full epoch 4 2023-12-07T07:32:48.623 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.619+0000 7fb60affd640 20 client.4121.objecter dump_active .. 0 homeless 2023-12-07T07:32:48.625 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:48.623+0000 7f753c845040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-07T07:32:48.632 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:48.630+0000 7f141e62b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-07T07:32:48.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.719+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb614043070 con 0x7fb61c0a8a00 2023-12-07T07:32:48.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.719+0000 7fb60affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:48.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.719+0000 7fb60affd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:48.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.788+0000 7fb620b5a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2e035cab-ae59-4d53-b46e-867d865bfc46", "id": 2}] 2023-12-07T07:32:48.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.788+0000 7fb620b5a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2e035cab-ae59-4d53-b46e-867d865bfc46", "id": 2}] 2023-12-07T07:32:48.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.788+0000 7fb620b5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:48.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.788+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2e035cab-ae59-4d53-b46e-867d865bfc46", "id": 2} v 0) v1 -- 0x7fb61c143750 con 0x7fb61c0a8a00 2023-12-07T07:32:48.835 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.833+0000 7fb60affd640 1 -- 172.21.15.154:0/236069033 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2e035cab-ae59-4d53-b46e-867d865bfc46", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb61c143750 con 0x7fb61c0a8a00 2023-12-07T07:32:48.835 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.833+0000 7fb60affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2e035cab-ae59-4d53-b46e-867d865bfc46", "id": 2}] 2023-12-07T07:32:48.835 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.833+0000 7fb60affd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:48.835 INFO:teuthology.orchestra.run.smithi154.stdout:2 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 10 librados: watch_flush enter 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 10 librados: watch_flush exit 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 20 client.4121.objecter shutdown clearing up homeless session... 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 10 client.4121.objecter successfully canceled tick 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 10 monclient: shutdown 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 msgr2=0x7fb5fc00afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb5fc00afd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb614032000 tx=0x7fb610009190 comp rx=0 tx=0).stop 2023-12-07T07:32:48.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 shutdown_connections 2023-12-07T07:32:48.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb61c0a8a00 0x7fb5fc00afd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5fc005f80 0x7fb5fc008410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 --2- 172.21.15.154:0/236069033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c13f080 0x7fb5fc0054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 >> 172.21.15.154:0/236069033 conn(0x7fb61c061a90 msgr2=0x7fb61c141290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:48.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 shutdown_connections 2023-12-07T07:32:48.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 -- 172.21.15.154:0/236069033 wait complete. 2023-12-07T07:32:48.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.834+0000 7fb620b5a640 1 librados: shutdown 2023-12-07T07:32:48.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new e1148e29-3789-483a-8463-ea1e1f964b39 3 2023-12-07T07:32:48.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.942+0000 7f9ec5101640 10 monclient: get_monmap_and_config 2023-12-07T07:32:48.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.942+0000 7f9ec5101640 10 monclient: build_initial_monmap 2023-12-07T07:32:48.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.942+0000 7f9ec5101640 10 monclient: monmap: 2023-12-07T07:32:48.944 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:48.944 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:48.943313+0000 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:48.943313+0000 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:48.945 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.947 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.942+0000 7f9ec5101640 1 Processor -- start 2023-12-07T07:32:48.947 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 1 -- start start 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient: init 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec013f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient(hunting): picked mon.noname-b con 0x7f9ec0071110 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ec0070490 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ec00afc80 0x7f9ec006c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient(hunting): picked mon.noname-c con 0x7f9ec00afc80 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec006bc20 con 0x7f9ec0070490 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec0071b00 con 0x7f9ec00afc80 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0071c40 con 0x7f9ec0071110 2023-12-07T07:32:48.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.943+0000 7f9ec5101640 10 monclient(hunting): authenticate will time out at 1263.300864s 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebf59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ec00afc80 0x7f9ec006c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebed9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec013f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebed9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec013f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.154:55614/0 (socket says 172.21.15.154:55614) 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebed9d640 1 -- 172.21.15.154:0/3512565377 learned_addr learned my addr 172.21.15.154:0/3512565377 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebed9d640 10 monclient(hunting): get_auth_request con 0x7f9ec0071110 auth_method 0 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebed9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebf59e640 10 monclient(hunting): get_auth_request con 0x7f9ec00afc80 auth_method 0 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebf59e640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.944+0000 7f9ebf59e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.945+0000 7f9ebed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.945+0000 7f9ebed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.945+0000 7f9ebed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.945+0000 7f9ebf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.945+0000 7f9ebf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.945+0000 7f9ebf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 10 monclient(hunting): handle_auth_done global_id 4124 payload 274 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 1 -- 172.21.15.154:0/3512565377 >> v2:172.21.15.154:3300/0 conn(0x7f9ec00afc80 msgr2=0x7f9ec006c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 1 --2- 172.21.15.154:0/3512565377 >> v2:172.21.15.154:3300/0 conn(0x7f9ec00afc80 0x7f9ec006c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 1 -- 172.21.15.154:0/3512565377 >> v1:172.21.15.154:6789/0 conn(0x7f9ec0070490 legacy=0x7f9ec013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 10 monclient: _finish_hunting 0 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 1 monclient: found mon.noname-b 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.946+0000 7f9ebed9d640 1 -- 172.21.15.154:0/3512565377 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec00714f0 con 0x7f9ec0071110 2023-12-07T07:32:48.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.948+0000 7f9ebed9d640 1 --2- 172.21.15.154:0/3512565377 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec013f0a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009870 tx=0x7f9eb402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23f4e19634399f8a server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:48.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 1 -- 172.21.15.154:0/3512565377 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4031640 con 0x7f9ec0071110 2023-12-07T07:32:48.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:48.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: dump: 2023-12-07T07:32:48.954 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: _finish_auth 0 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: _check_auth_tickets 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 1 -- 172.21.15.154:0/3512565377 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb40317a0 con 0x7f9ec0071110 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ec5101640 5 monclient: authenticate success, global_id 4124 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ec5101640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ec5101640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 1 -- 172.21.15.154:0/3512565377 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4031b30 con 0x7f9ec0071110 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ec5101640 4 set_mon_vals no callback set 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.951+0000 7f9ebdd9b640 10 monclient: dump: 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.956 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 10 monclient: shutdown 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 -- 172.21.15.154:0/3512565377 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 msgr2=0x7f9ec013f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 --2- 172.21.15.154:0/3512565377 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec013f0a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009870 tx=0x7f9eb402b8f0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 -- 172.21.15.154:0/3512565377 shutdown_connections 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 --2- 172.21.15.154:0/3512565377 >> v2:172.21.15.154:3300/0 conn(0x7f9ec00afc80 0x7f9ec006c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 --2- 172.21.15.154:0/3512565377 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec013f0a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 -- 172.21.15.154:0/3512565377 >> 172.21.15.154:0/3512565377 conn(0x7f9ec0061970 msgr2=0x7f9ec00b5910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 -- 172.21.15.154:0/3512565377 shutdown_connections 2023-12-07T07:32:48.957 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.952+0000 7f9ec5101640 1 -- 172.21.15.154:0/3512565377 wait complete. 2023-12-07T07:32:48.960 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 10 monclient: build_initial_monmap 2023-12-07T07:32:48.960 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 10 monclient: monmap: 2023-12-07T07:32:48.960 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 1 librados: starting msgr at 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 1 librados: starting objecter 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 1 Processor -- start 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 1 -- start start 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 1 librados: setting wanted keys 2023-12-07T07:32:48.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 1 librados: calling monclient init 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.953+0000 7f9ec5101640 10 monclient: init 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec0070490 0x7f9ec00af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): picked mon.noname-b con 0x7f9ec0070490 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec0139250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): picked mon.noname-c con 0x7f9ec0071110 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ec00afc80 0x7f9ec0142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): picked mon.noname-a con 0x7f9ec00afc80 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01421f0 con 0x7f9ec00afc80 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0138870 con 0x7f9ec0070490 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:48.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0138b50 con 0x7f9ec0071110 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ec5101640 10 monclient(hunting): authenticate will time out at 1263.311688s 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.954+0000 7f9ebed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec0070490 0x7f9ec00af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec0139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ec00afc80 0x7f9ec0142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec0070490 0x7f9ec00af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:48432/0 (socket says 172.21.15.154:48432) 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 1 -- 172.21.15.154:0/1805451451 learned_addr learned my addr 172.21.15.154:0/1805451451 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 10 monclient(hunting): get_auth_request con 0x7f9ec0070490 auth_method 0 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 10 monclient(hunting): get_auth_request con 0x7f9ec0071110 auth_method 0 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 10 monclient(hunting): get_auth_request con 0x7f9ec00afc80 auth_method 0 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.955+0000 7f9ebf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:48.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 10 monclient(hunting): handle_auth_done global_id 4127 payload 824 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 1 -- 172.21.15.154:0/1805451451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec0070490 msgr2=0x7f9ec00af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 1 --2- 172.21.15.154:0/1805451451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec0070490 0x7f9ec00af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 1 -- 172.21.15.154:0/1805451451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ec00afc80 msgr2=0x7f9ec0142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 1 --2- 172.21.15.154:0/1805451451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ec00afc80 0x7f9ec0142790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 10 monclient: _finish_hunting 0 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 1 monclient: found mon.noname-c 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.958+0000 7f9ebe59c640 1 -- 172.21.15.154:0/1805451451 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb40094d0 con 0x7f9ec0071110 2023-12-07T07:32:48.966 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.960+0000 7f9ebe59c640 1 --2- 172.21.15.154:0/1805451451 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec0139250 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000c390 tx=0x7f9eb000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:48.966 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.960+0000 7f9ea77fe640 10 client.?.objecter ms_handle_connect 0x7f9ec0071110 2023-12-07T07:32:48.966 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.960+0000 7f9ea77fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:48.966 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 1 -- 172.21.15.154:0/1805451451 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00052d0 con 0x7f9ec0071110 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: dump: 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: _finish_auth 0 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: _check_auth_tickets 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 1 -- 172.21.15.154:0/1805451451 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0005430 con 0x7f9ec0071110 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 1 -- 172.21.15.154:0/1805451451 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0003830 con 0x7f9ec0071110 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ea77fe640 10 monclient: dump: 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:48.968 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 5 monclient: authenticate success, global_id 4127 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 10 monclient: _renew_subs 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec0141090 con 0x7f9ec0071110 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ebd59a640 4 set_mon_vals no callback set 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 10 monclient: _renew_subs 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec0142eb0 con 0x7f9ec0071110 2023-12-07T07:32:48.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.964+0000 7f9ec5101640 1 librados: init done 2023-12-07T07:32:48.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.965+0000 7f9ec5101640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:48.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.965+0000 7f9ec5101640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:48.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.965+0000 7f9ec5101640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:48.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.965+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec0075b60 con 0x7f9ec0071110 2023-12-07T07:32:48.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.967+0000 7f9ea77fe640 1 -- 172.21.15.154:0/1805451451 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9eb0003c90 con 0x7f9ec0071110 2023-12-07T07:32:48.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.967+0000 7f9ea77fe640 1 -- 172.21.15.154:0/1805451451 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f9eb0011da0 con 0x7f9ec0071110 2023-12-07T07:32:48.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.967+0000 7f9ea77fe640 10 client.4127.objecter ms_dispatch 0x7f9ec0000b90 osd_map(5..5 src has 1..5) v4 2023-12-07T07:32:48.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.967+0000 7f9ea77fe640 3 client.4127.objecter handle_osd_map got epochs [5,5] > 0 2023-12-07T07:32:48.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.967+0000 7f9ea77fe640 3 client.4127.objecter handle_osd_map decoding full epoch 5 2023-12-07T07:32:48.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:48.967+0000 7f9ea77fe640 20 client.4127.objecter dump_active .. 0 homeless 2023-12-07T07:32:49.025 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:49.023+0000 7f141e62b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-07T07:32:49.038 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:49.036+0000 7f753c845040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-07T07:32:49.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.066+0000 7f9ea77fe640 1 -- 172.21.15.154:0/1805451451 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9eb0017070 con 0x7f9ec0071110 2023-12-07T07:32:49.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.066+0000 7f9ea77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:49.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.066+0000 7f9ea77fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:49.074 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:49.072+0000 7f141e62b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-07T07:32:49.086 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:49.084+0000 7f753c845040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-07T07:32:49.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.141+0000 7f9ec5101640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e1148e29-3789-483a-8463-ea1e1f964b39", "id": 3}] 2023-12-07T07:32:49.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.141+0000 7f9ec5101640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e1148e29-3789-483a-8463-ea1e1f964b39", "id": 3}] 2023-12-07T07:32:49.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.141+0000 7f9ec5101640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:49.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.141+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e1148e29-3789-483a-8463-ea1e1f964b39", "id": 3} v 0) v1 -- 0x7f9ec006af70 con 0x7f9ec0071110 2023-12-07T07:32:49.182 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:49.180+0000 7f141e62b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-07T07:32:49.193 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:49.190+0000 7f753c845040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-07T07:32:49.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.206+0000 7f9ea77fe640 1 -- 172.21.15.154:0/1805451451 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1148e29-3789-483a-8463-ea1e1f964b39", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9eb0011590 con 0x7f9ec0071110 2023-12-07T07:32:49.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.206+0000 7f9ea77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e1148e29-3789-483a-8463-ea1e1f964b39", "id": 3}] 2023-12-07T07:32:49.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.206+0000 7f9ea77fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:49.208 INFO:teuthology.orchestra.run.smithi154.stdout:3 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 10 librados: watch_flush enter 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 10 librados: watch_flush exit 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 20 client.4127.objecter shutdown clearing up homeless session... 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 10 client.4127.objecter successfully canceled tick 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 10 monclient: shutdown 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 msgr2=0x7f9ec0139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 --2- 172.21.15.154:0/1805451451 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec0139250 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000c390 tx=0x7f9eb000c850 comp rx=0 tx=0).stop 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 shutdown_connections 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 --2- 172.21.15.154:0/1805451451 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ec00afc80 0x7f9ec0142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 --2- 172.21.15.154:0/1805451451 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ec0071110 0x7f9ec0139250 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 --2- 172.21.15.154:0/1805451451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec0070490 0x7f9ec00af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 >> 172.21.15.154:0/1805451451 conn(0x7f9ec0061970 msgr2=0x7f9ec00b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:49.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 shutdown_connections 2023-12-07T07:32:49.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 -- 172.21.15.154:0/1805451451 wait complete. 2023-12-07T07:32:49.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.207+0000 7f9ec5101640 1 librados: shutdown 2023-12-07T07:32:49.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new f525c573-e372-4e52-ab23-48a4a0905272 4 2023-12-07T07:32:49.281 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:49.279+0000 7f141e62b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-07T07:32:49.295 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:49.293+0000 7f753c845040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-07T07:32:49.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.318+0000 7f8006933640 10 monclient: get_monmap_and_config 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.318+0000 7f8006933640 10 monclient: build_initial_monmap 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.318+0000 7f8006933640 10 monclient: monmap: 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:49.319791+0000 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:49.319791+0000 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:49.321 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.319+0000 7f8006933640 1 Processor -- start 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.319+0000 7f8006933640 1 -- start start 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.319+0000 7f8006933640 10 monclient: init 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80000a8a00 0x7f80000a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient(hunting): picked mon.noname-c con 0x7f80000a8a00 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80000b2260 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f800013f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient(hunting): picked mon.noname-b con 0x7f800013f080 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:49.324 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80000aa970 con 0x7f80000b2260 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80001421b0 con 0x7f80000a8a00 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80000aac50 con 0x7f800013f080 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f8006933640 10 monclient(hunting): authenticate will time out at 1263.677580s 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.320+0000 7f7ffffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80000a8a00 0x7f80000a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7fff7fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f80000b2260 0x7f80001425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.154:45722/0 (socket says 172.21.15.154:45722) 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7fff7fe640 1 -- 172.21.15.154:0/832869922 learned_addr learned my addr 172.21.15.154:0/832869922 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f8004ed0640 1 --2- 172.21.15.154:0/832869922 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f800013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7ffffff640 10 monclient(hunting): get_auth_request con 0x7f80000a8a00 auth_method 0 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7ffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7ffffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.325 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7ffffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f8004ed0640 10 monclient(hunting): get_auth_request con 0x7f800013f080 auth_method 0 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f8004ed0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f8004ed0640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f8004ed0640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7ffffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7ffffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.321+0000 7f7ffffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.322+0000 7f8004ed0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.322+0000 7f8004ed0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.322+0000 7f8004ed0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffeffd640 1 -- 172.21.15.154:0/832869922 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889023082 0 0) 0x7f80000aa970 con 0x7f80000b2260 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffeffd640 10 monclient(hunting): _negotiate 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffeffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffeffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffeffd640 10 monclient(hunting): my global_id is 4122 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffeffd640 1 -- 172.21.15.154:0/832869922 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8003520 con 0x7f80000b2260 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2023-12-07T07:32:49.327 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 1 -- 172.21.15.154:0/832869922 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 msgr2=0x7f800013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 1 --2- 172.21.15.154:0/832869922 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f800013f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 1 -- 172.21.15.154:0/832869922 >> v1:172.21.15.97:6789/0 conn(0x7f80000b2260 legacy=0x7f80001425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 10 monclient: _finish_hunting 0 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 1 monclient: found mon.noname-c 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.323+0000 7f7ffffff640 1 -- 172.21.15.154:0/832869922 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80000aaf30 con 0x7f80000a8a00 2023-12-07T07:32:49.328 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.325+0000 7f7ffffff640 1 --2- 172.21.15.154:0/832869922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000a8a00 0x7f80000a8de0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40097a0 tx=0x7f7ff402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=838840da3e949f7a server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 1 -- 172.21.15.154:0/832869922 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff402f050 con 0x7f80000a8a00 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: dump: 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.331 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: _finish_auth 0 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 1 -- 172.21.15.154:0/832869922 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff4031700 con 0x7f80000a8a00 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 1 -- 172.21.15.154:0/832869922 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4031a90 con 0x7f80000a8a00 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f8006933640 5 monclient: authenticate success, global_id 4135 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f8006933640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f8006933640 4 set_mon_vals no callback set 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.329+0000 7f7ffeffd640 10 monclient: dump: 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.333 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 10 monclient: shutdown 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 -- 172.21.15.154:0/832869922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000a8a00 msgr2=0x7f80000a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 --2- 172.21.15.154:0/832869922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000a8a00 0x7f80000a8de0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40097a0 tx=0x7f7ff402b7a0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 -- 172.21.15.154:0/832869922 shutdown_connections 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 --2- 172.21.15.154:0/832869922 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f800013f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 --2- 172.21.15.154:0/832869922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000a8a00 0x7f80000a8de0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 -- 172.21.15.154:0/832869922 >> 172.21.15.154:0/832869922 conn(0x7f8000061a90 msgr2=0x7f80000b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 -- 172.21.15.154:0/832869922 shutdown_connections 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.330+0000 7f8006933640 1 -- 172.21.15.154:0/832869922 wait complete. 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.331+0000 7f8006933640 10 monclient: build_initial_monmap 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.331+0000 7f8006933640 10 monclient: monmap: 2023-12-07T07:32:49.342 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.331+0000 7f8006933640 1 librados: starting msgr at 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.331+0000 7f8006933640 1 librados: starting objecter 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.331+0000 7f8006933640 1 Processor -- start 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 -- start start 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 librados: setting wanted keys 2023-12-07T07:32:49.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 librados: calling monclient init 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient: init 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80000a8a00 0x7f8000139990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient(hunting): picked mon.noname-a con 0x7f80000a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000b2260 0x7f80000b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient(hunting): picked mon.noname-b con 0x7f80000b2260 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f80001d7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient(hunting): picked mon.noname-c con 0x7f800013f080 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80000b8350 con 0x7f80000a8a00 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800013aaa0 con 0x7f80000b2260 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.332+0000 7f8006933640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800013ad80 con 0x7f800013f080 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f8006933640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f8006933640 10 monclient(hunting): authenticate will time out at 1263.690201s 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7fff7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000b2260 0x7f80000b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f8004ed0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f80001d7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80000a8a00 0x7f8000139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7fff7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000b2260 0x7f80000b7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:48448/0 (socket says 172.21.15.154:48448) 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7fff7fe640 1 -- 172.21.15.154:0/484752046 learned_addr learned my addr 172.21.15.154:0/484752046 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7fff7fe640 10 monclient(hunting): get_auth_request con 0x7f80000b2260 auth_method 0 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7fff7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7fff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f8004ed0640 10 monclient(hunting): get_auth_request con 0x7f800013f080 auth_method 0 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f8004ed0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f8004ed0640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f8004ed0640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7ffffff640 10 monclient(hunting): get_auth_request con 0x7f80000a8a00 auth_method 0 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7ffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7ffffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.333+0000 7f7ffffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f7fff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f7fff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f7fff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f7ffffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f7ffffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f7ffffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f8004ed0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f8004ed0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.334+0000 7f8004ed0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 1 -- 172.21.15.154:0/484752046 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 msgr2=0x7f80001d7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f80001d7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 1 -- 172.21.15.154:0/484752046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80000a8a00 msgr2=0x7f8000139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80000a8a00 0x7f8000139990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 1 monclient: found mon.noname-b 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7fff7fe640 1 -- 172.21.15.154:0/484752046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff4009400 con 0x7f80000b2260 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.337+0000 7f7ffffff640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80000a8a00 0x7f8000139990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.339+0000 7f7fff7fe640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000b2260 0x7f80000b7e10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000c400 tx=0x7f7ff000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.339+0000 7f7ffcff9640 10 client.?.objecter ms_handle_connect 0x7f80000b2260 2023-12-07T07:32:49.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.339+0000 7f7ffcff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 1 -- 172.21.15.154:0/484752046 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0005ce0 con 0x7f80000b2260 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: dump: 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.349 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: _finish_auth 0 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: _check_auth_tickets 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 5 monclient: authenticate success, global_id 4138 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 1 -- 172.21.15.154:0/484752046 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff0005e60 con 0x7f80000b2260 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 1 -- 172.21.15.154:0/484752046 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00072f0 con 0x7f80000b2260 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 10 monclient: _renew_subs 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80001d70f0 con 0x7f80000b2260 2023-12-07T07:32:49.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffcff9640 10 monclient: dump: 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f7ffe7fc640 4 set_mon_vals no callback set 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 10 monclient: _renew_subs 2023-12-07T07:32:49.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800013b660 con 0x7f80000b2260 2023-12-07T07:32:49.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.342+0000 7f8006933640 1 librados: init done 2023-12-07T07:32:49.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.343+0000 7f8006933640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:49.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.343+0000 7f8006933640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:49.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.343+0000 7f8006933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.343+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8000067ac0 con 0x7f80000b2260 2023-12-07T07:32:49.424 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:49.422+0000 7f141e62b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-07T07:32:49.444 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:49.442+0000 7f753c845040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-07T07:32:49.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 1 -- 172.21.15.154:0/484752046 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7ff001c440 con 0x7f80000b2260 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 1 -- 172.21.15.154:0/484752046 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f7ff001b8c0 con 0x7f80000b2260 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 10 client.4138.objecter ms_dispatch 0x7f8000000b90 osd_map(6..6 src has 1..6) v4 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 20 client.4138.objecter dump_active .. 0 homeless 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 1 -- 172.21.15.154:0/484752046 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7ff0017070 con 0x7f80000b2260 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:49.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.447+0000 7f7ffcff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:49.466 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:49.464+0000 7f141e62b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-07T07:32:49.488 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:49.486+0000 7f753c845040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-07T07:32:49.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.520+0000 7f8006933640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f525c573-e372-4e52-ab23-48a4a0905272", "id": 4}] 2023-12-07T07:32:49.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.520+0000 7f8006933640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f525c573-e372-4e52-ab23-48a4a0905272", "id": 4}] 2023-12-07T07:32:49.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.520+0000 7f8006933640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.520+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f525c573-e372-4e52-ab23-48a4a0905272", "id": 4} v 0) v1 -- 0x7f8000075b60 con 0x7f80000b2260 2023-12-07T07:32:49.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.589+0000 7f7ffcff9640 1 -- 172.21.15.154:0/484752046 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f525c573-e372-4e52-ab23-48a4a0905272", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7ff001b500 con 0x7f80000b2260 2023-12-07T07:32:49.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.589+0000 7f7ffcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f525c573-e372-4e52-ab23-48a4a0905272", "id": 4}] 2023-12-07T07:32:49.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.589+0000 7f7ffcff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:49.591 INFO:teuthology.orchestra.run.smithi154.stdout:4 2023-12-07T07:32:49.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 10 librados: watch_flush enter 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 10 librados: watch_flush exit 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 20 client.4138.objecter shutdown clearing up homeless session... 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 10 client.4138.objecter successfully canceled tick 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 10 monclient: shutdown 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000b2260 msgr2=0x7f80000b7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000b2260 0x7f80000b7e10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000c400 tx=0x7f7ff000c8c0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 shutdown_connections 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f800013f080 0x7f80001d7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000b2260 0x7f80000b7e10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 --2- 172.21.15.154:0/484752046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f80000a8a00 0x7f8000139990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 >> 172.21.15.154:0/484752046 conn(0x7f8000061a90 msgr2=0x7f800013fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:49.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 shutdown_connections 2023-12-07T07:32:49.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 -- 172.21.15.154:0/484752046 wait complete. 2023-12-07T07:32:49.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.590+0000 7f8006933640 1 librados: shutdown 2023-12-07T07:32:49.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 4dfbc265-0117-4a4e-95f8-433766bd4aab 5 2023-12-07T07:32:49.612 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:49.610+0000 7f141e62b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-07T07:32:49.640 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:49.638+0000 7f753c845040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-07T07:32:49.701 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.699+0000 7f9f92761640 10 monclient: get_monmap_and_config 2023-12-07T07:32:49.701 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.699+0000 7f9f92761640 10 monclient: build_initial_monmap 2023-12-07T07:32:49.701 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.699+0000 7f9f92761640 10 monclient: monmap: 2023-12-07T07:32:49.701 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:49.701 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.701 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:49.700120+0000 2023-12-07T07:32:49.701 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:49.700120+0000 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:49.702 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.699+0000 7f9f92761640 1 Processor -- start 2023-12-07T07:32:49.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 1 -- start start 2023-12-07T07:32:49.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient: init 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9f8c0a8a00 0x7f9f8c0a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient(hunting): picked mon.noname-c con 0x7f9f8c0a8a00 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f8c0b2260 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c13f080 0x7f9f8c13f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient(hunting): picked mon.noname-b con 0x7f9f8c13f080 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c0aa970 con 0x7f9f8c0b2260 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8c1421b0 con 0x7f9f8c0a8a00 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0aac50 con 0x7f9f8c13f080 2023-12-07T07:32:49.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.700+0000 7f9f92761640 10 monclient(hunting): authenticate will time out at 1264.057906s 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f8bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9f8c0a8a00 0x7f9f8c0a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f8b7fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9f8c0b2260 0x7f9f8c1425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.154:45738/0 (socket says 172.21.15.154:45738) 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f8b7fe640 1 -- 172.21.15.154:0/1699894250 learned_addr learned my addr 172.21.15.154:0/1699894250 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f90cfe640 1 --2- 172.21.15.154:0/1699894250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c13f080 0x7f9f8c13f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f8bfff640 10 monclient(hunting): get_auth_request con 0x7f9f8c0a8a00 auth_method 0 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f8bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f90cfe640 10 monclient(hunting): get_auth_request con 0x7f9f8c13f080 auth_method 0 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f90cfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f90cfe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.701+0000 7f9f90cfe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.702+0000 7f9f8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.702+0000 7f9f8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.702+0000 7f9f8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.702+0000 7f9f90cfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.702+0000 7f9f90cfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.702+0000 7f9f90cfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 1 -- 172.21.15.154:0/1699894250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c13f080 msgr2=0x7f9f8c13f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 1 --2- 172.21.15.154:0/1699894250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c13f080 0x7f9f8c13f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 1 -- 172.21.15.154:0/1699894250 >> v1:172.21.15.97:6789/0 conn(0x7f9f8c0b2260 legacy=0x7f9f8c1425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 10 monclient: _finish_hunting 0 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 1 monclient: found mon.noname-c 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.703+0000 7f9f8bfff640 1 -- 172.21.15.154:0/1699894250 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0aaf30 con 0x7f9f8c0a8a00 2023-12-07T07:32:49.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.705+0000 7f9f8bfff640 1 --2- 172.21.15.154:0/1699894250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0a8a00 0x7f9f8c0a8de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0097a0 tx=0x7f9f7c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb53af15f8cbabb3 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:49.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 1 -- 172.21.15.154:0/1699894250 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c02f050 con 0x7f9f8c0a8a00 2023-12-07T07:32:49.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: dump: 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.714 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: _finish_auth 0 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f92761640 5 monclient: authenticate success, global_id 4141 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 1 -- 172.21.15.154:0/1699894250 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c031700 con 0x7f9f8c0a8a00 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f92761640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f92761640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f92761640 4 set_mon_vals no callback set 2023-12-07T07:32:49.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 1 -- 172.21.15.154:0/1699894250 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c031a90 con 0x7f9f8c0a8a00 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.709+0000 7f9f8affd640 10 monclient: dump: 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 10 monclient: shutdown 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 -- 172.21.15.154:0/1699894250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0a8a00 msgr2=0x7f9f8c0a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 --2- 172.21.15.154:0/1699894250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0a8a00 0x7f9f8c0a8de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0097a0 tx=0x7f9f7c02b7a0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 -- 172.21.15.154:0/1699894250 shutdown_connections 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 --2- 172.21.15.154:0/1699894250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c13f080 0x7f9f8c13f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 --2- 172.21.15.154:0/1699894250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0a8a00 0x7f9f8c0a8de0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 -- 172.21.15.154:0/1699894250 >> 172.21.15.154:0/1699894250 conn(0x7f9f8c061a90 msgr2=0x7f9f8c0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 -- 172.21.15.154:0/1699894250 shutdown_connections 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.710+0000 7f9f92761640 1 -- 172.21.15.154:0/1699894250 wait complete. 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 10 monclient: build_initial_monmap 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 10 monclient: monmap: 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:49.717 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.718 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:49.718 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:49.718 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:49.718 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 1 librados: starting msgr at 2023-12-07T07:32:49.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 1 librados: starting objecter 2023-12-07T07:32:49.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 1 Processor -- start 2023-12-07T07:32:49.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 1 -- start start 2023-12-07T07:32:49.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 1 librados: setting wanted keys 2023-12-07T07:32:49.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 1 librados: calling monclient init 2023-12-07T07:32:49.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.711+0000 7f9f92761640 10 monclient: init 2023-12-07T07:32:49.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:49.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:32:49.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c0a8a00 0x7f9f8c1432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): picked mon.noname-c con 0x7f9f8c0a8a00 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:49.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0b2260 0x7f9f8c1d76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): picked mon.noname-b con 0x7f9f8c0b2260 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f8c13f080 0x7f9f8c137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): picked mon.noname-a con 0x7f9f8c13f080 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c1335d0 con 0x7f9f8c13f080 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c1d7230 con 0x7f9f8c0b2260 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:49.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c1d6ec0 con 0x7f9f8c0a8a00 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.712+0000 7f9f92761640 10 monclient(hunting): authenticate will time out at 1264.069742s 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0b2260 0x7f9f8c1d76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8bfff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c0a8a00 0x7f9f8c1432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f90cfe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f8c13f080 0x7f9f8c137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0b2260 0x7f9f8c1d76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:48480/0 (socket says 172.21.15.154:48480) 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 1 -- 172.21.15.154:0/337317750 learned_addr learned my addr 172.21.15.154:0/337317750 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 10 monclient(hunting): get_auth_request con 0x7f9f8c0b2260 auth_method 0 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f90cfe640 10 monclient(hunting): get_auth_request con 0x7f9f8c13f080 auth_method 0 2023-12-07T07:32:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f90cfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f90cfe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f90cfe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8bfff640 10 monclient(hunting): get_auth_request con 0x7f9f8c0a8a00 auth_method 0 2023-12-07T07:32:49.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.713+0000 7f9f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.714+0000 7f9f8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.714+0000 7f9f90cfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.714+0000 7f9f90cfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.714+0000 7f9f90cfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.714+0000 7f9f8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.714+0000 7f9f8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.714+0000 7f9f8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 1 -- 172.21.15.154:0/337317750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c0a8a00 msgr2=0x7f9f8c1432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 1 --2- 172.21.15.154:0/337317750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c0a8a00 0x7f9f8c1432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 1 -- 172.21.15.154:0/337317750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f8c13f080 msgr2=0x7f9f8c137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 1 --2- 172.21.15.154:0/337317750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f8c13f080 0x7f9f8c137270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 1 monclient: found mon.noname-b 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.716+0000 7f9f8b7fe640 1 -- 172.21.15.154:0/337317750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c009400 con 0x7f9f8c0b2260 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.718+0000 7f9f8b7fe640 1 --2- 172.21.15.154:0/337317750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0b2260 0x7f9f8c1d76c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000c470 tx=0x7f9f80001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.718+0000 7f9f88ff9640 10 client.?.objecter ms_handle_connect 0x7f9f8c0b2260 2023-12-07T07:32:49.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.718+0000 7f9f88ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:49.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 1 -- 172.21.15.154:0/337317750 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80005ce0 con 0x7f9f8c0b2260 2023-12-07T07:32:49.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:49.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: dump: 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: _finish_auth 0 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 1 -- 172.21.15.154:0/337317750 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f80005e40 con 0x7f9f8c0b2260 2023-12-07T07:32:49.731 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 1 -- 172.21.15.154:0/337317750 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f800072f0 con 0x7f9f8c0b2260 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f88ff9640 10 monclient: dump: 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 5 monclient: authenticate success, global_id 4144 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 10 monclient: _renew_subs 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f8a7fc640 4 set_mon_vals no callback set 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c1452d0 con 0x7f9f8c0b2260 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 10 monclient: _renew_subs 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.722+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c1d7de0 con 0x7f9f8c0b2260 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.723+0000 7f9f92761640 1 librados: init done 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.724+0000 7f9f92761640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.724+0000 7f9f92761640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.724+0000 7f9f92761640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.724+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8c1d7080 con 0x7f9f8c0b2260 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.725+0000 7f9f88ff9640 1 -- 172.21.15.154:0/337317750 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9f8001c440 con 0x7f9f8c0b2260 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.725+0000 7f9f88ff9640 1 -- 172.21.15.154:0/337317750 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f9f80012980 con 0x7f9f8c0b2260 2023-12-07T07:32:49.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.725+0000 7f9f88ff9640 10 client.4144.objecter ms_dispatch 0x7f9f8c000b90 osd_map(7..7 src has 1..7) v4 2023-12-07T07:32:49.734 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.725+0000 7f9f88ff9640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2023-12-07T07:32:49.734 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.725+0000 7f9f88ff9640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2023-12-07T07:32:49.734 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.725+0000 7f9f88ff9640 20 client.4144.objecter dump_active .. 0 homeless 2023-12-07T07:32:49.830 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.828+0000 7f9f88ff9640 1 -- 172.21.15.154:0/337317750 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9f8c1d7de0 con 0x7f9f8c0b2260 2023-12-07T07:32:49.830 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.828+0000 7f9f88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:49.830 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.828+0000 7f9f88ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:49.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.900+0000 7f9f92761640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4dfbc265-0117-4a4e-95f8-433766bd4aab", "id": 5}] 2023-12-07T07:32:49.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.900+0000 7f9f92761640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4dfbc265-0117-4a4e-95f8-433766bd4aab", "id": 5}] 2023-12-07T07:32:49.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.900+0000 7f9f92761640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:49.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.900+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4dfbc265-0117-4a4e-95f8-433766bd4aab", "id": 5} v 0) v1 -- 0x7f9f8c067ac0 con 0x7f9f8c0b2260 2023-12-07T07:32:49.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.970+0000 7f9f88ff9640 1 -- 172.21.15.154:0/337317750 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4dfbc265-0117-4a4e-95f8-433766bd4aab", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9f80017070 con 0x7f9f8c0b2260 2023-12-07T07:32:49.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.970+0000 7f9f88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4dfbc265-0117-4a4e-95f8-433766bd4aab", "id": 5}] 2023-12-07T07:32:49.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.970+0000 7f9f88ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:49.973 INFO:teuthology.orchestra.run.smithi154.stdout:5 2023-12-07T07:32:49.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 10 librados: watch_flush enter 2023-12-07T07:32:49.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 10 librados: watch_flush exit 2023-12-07T07:32:49.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 20 client.4144.objecter shutdown clearing up homeless session... 2023-12-07T07:32:49.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 10 client.4144.objecter successfully canceled tick 2023-12-07T07:32:49.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 10 monclient: shutdown 2023-12-07T07:32:49.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:49.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0b2260 msgr2=0x7f9f8c1d76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 1 --2- 172.21.15.154:0/337317750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0b2260 0x7f9f8c1d76c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000c470 tx=0x7f9f80001e70 comp rx=0 tx=0).stop 2023-12-07T07:32:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 shutdown_connections 2023-12-07T07:32:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.971+0000 7f9f92761640 1 --2- 172.21.15.154:0/337317750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f8c13f080 0x7f9f8c137270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.972+0000 7f9f92761640 1 --2- 172.21.15.154:0/337317750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f8c0b2260 0x7f9f8c1d76c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.972+0000 7f9f92761640 1 --2- 172.21.15.154:0/337317750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9f8c0a8a00 0x7f9f8c1432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.972+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 >> 172.21.15.154:0/337317750 conn(0x7f9f8c061a90 msgr2=0x7f9f8c1424d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.972+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 shutdown_connections 2023-12-07T07:32:49.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.972+0000 7f9f92761640 1 -- 172.21.15.154:0/337317750 wait complete. 2023-12-07T07:32:49.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:49.972+0000 7f9f92761640 1 librados: shutdown 2023-12-07T07:32:49.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd 6 2023-12-07T07:32:50.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.124+0000 7fc69aa2b640 10 monclient: get_monmap_and_config 2023-12-07T07:32:50.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.124+0000 7fc69aa2b640 10 monclient: build_initial_monmap 2023-12-07T07:32:50.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.124+0000 7fc69aa2b640 10 monclient: monmap: 2023-12-07T07:32:50.126 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:50.126 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.126 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:50.125093+0000 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:50.125093+0000 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:50.127 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.124+0000 7fc69aa2b640 1 Processor -- start 2023-12-07T07:32:50.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.124+0000 7fc69aa2b640 1 -- start start 2023-12-07T07:32:50.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.124+0000 7fc69aa2b640 10 monclient: init 2023-12-07T07:32:50.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6940a8a00 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6940b2260 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc6940a8de0 0x7fc69413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient(hunting): picked mon.noname-c con 0x7fc6940a8de0 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6940aa970 con 0x7fc6940b2260 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6940aac50 con 0x7fc6940a8a00 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6941421b0 con 0x7fc6940a8de0 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.125+0000 7fc69aa2b640 10 monclient(hunting): authenticate will time out at 1264.482802s 2023-12-07T07:32:50.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.131+0000 7fc698fc8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc6940a8de0 0x7fc69413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.131+0000 7fc693fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc6940a8a00 0x7fc6941425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:60660/0 (socket says 172.21.15.154:60660) 2023-12-07T07:32:50.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.131+0000 7fc693fff640 1 -- 172.21.15.154:0/3056578139 learned_addr learned my addr 172.21.15.154:0/3056578139 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-12-07T07:32:50.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.133+0000 7fc698fc8640 10 monclient(hunting): get_auth_request con 0x7fc6940a8de0 auth_method 0 2023-12-07T07:32:50.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.133+0000 7fc698fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.133+0000 7fc698fc8640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.133+0000 7fc698fc8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.134+0000 7fc698fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.134+0000 7fc698fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.134+0000 7fc698fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 1 -- 172.21.15.154:0/3056578139 >> v1:172.21.15.154:6789/0 conn(0x7fc6940a8a00 legacy=0x7fc6941425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 1 -- 172.21.15.154:0/3056578139 >> v1:172.21.15.97:6789/0 conn(0x7fc6940b2260 legacy=0x7fc69413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 10 monclient: _finish_hunting 0 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 1 monclient: found mon.noname-c 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.138+0000 7fc698fc8640 1 -- 172.21.15.154:0/3056578139 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6940aaf30 con 0x7fc6940a8de0 2023-12-07T07:32:50.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.141+0000 7fc698fc8640 1 --2- 172.21.15.154:0/3056578139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8de0 0x7fc69413f790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009d90 tx=0x7fc68c02bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93720d237bb3598c server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:50.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 1 -- 172.21.15.154:0/3056578139 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c02cb40 con 0x7fc6940a8de0 2023-12-07T07:32:50.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:32:50.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: dump: 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: _finish_auth 0 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 1 -- 172.21.15.154:0/3056578139 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c02cca0 con 0x7fc6940a8de0 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc69aa2b640 5 monclient: authenticate success, global_id 4147 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc69aa2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc69aa2b640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 1 -- 172.21.15.154:0/3056578139 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c034660 con 0x7fc6940a8de0 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc69aa2b640 4 set_mon_vals no callback set 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.148+0000 7fc692ffd640 10 monclient: dump: 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.157 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 10 monclient: shutdown 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 1 -- 172.21.15.154:0/3056578139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8de0 msgr2=0x7fc69413f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 1 --2- 172.21.15.154:0/3056578139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8de0 0x7fc69413f790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009d90 tx=0x7fc68c02bd90 comp rx=0 tx=0).stop 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 1 -- 172.21.15.154:0/3056578139 shutdown_connections 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 1 --2- 172.21.15.154:0/3056578139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8de0 0x7fc69413f790 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 1 -- 172.21.15.154:0/3056578139 >> 172.21.15.154:0/3056578139 conn(0x7fc694061a90 msgr2=0x7fc6940b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 1 -- 172.21.15.154:0/3056578139 shutdown_connections 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.149+0000 7fc69aa2b640 1 -- 172.21.15.154:0/3056578139 wait complete. 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 10 monclient: build_initial_monmap 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 10 monclient: monmap: 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 1 librados: starting msgr at 2023-12-07T07:32:50.159 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 1 librados: starting objecter 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 1 Processor -- start 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 1 -- start start 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 1 librados: setting wanted keys 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 1 librados: calling monclient init 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.150+0000 7fc69aa2b640 10 monclient: init 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8a00 0x7fc694134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): picked mon.noname-b con 0x7fc6940a8a00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6940a8de0 0x7fc694145de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): picked mon.noname-c con 0x7fc6940a8de0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6940b2260 0x7fc6941364b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): picked mon.noname-a con 0x7fc6940b2260 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc694146320 con 0x7fc6940b2260 2023-12-07T07:32:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc694135ad0 con 0x7fc6940a8a00 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc694135db0 con 0x7fc6940a8de0 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc69aa2b640 10 monclient(hunting): authenticate will time out at 1264.508691s 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.151+0000 7fc693fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8a00 0x7fc694134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc6937fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6940a8de0 0x7fc694145de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc698fc8640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6940b2260 0x7fc6941364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8a00 0x7fc694134ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:48502/0 (socket says 172.21.15.154:48502) 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 1 -- 172.21.15.154:0/4294447880 learned_addr learned my addr 172.21.15.154:0/4294447880 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 10 monclient(hunting): get_auth_request con 0x7fc6940a8a00 auth_method 0 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.161 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc6937fe640 10 monclient(hunting): get_auth_request con 0x7fc6940a8de0 auth_method 0 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc6937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc6937fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc6937fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc698fc8640 10 monclient(hunting): get_auth_request con 0x7fc6940b2260 auth_method 0 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc698fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc698fc8640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc698fc8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.152+0000 7fc693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.153+0000 7fc698fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.153+0000 7fc698fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.153+0000 7fc698fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.153+0000 7fc6937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.153+0000 7fc6937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.153+0000 7fc6937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 1 -- 172.21.15.154:0/4294447880 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6940a8de0 msgr2=0x7fc694145de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 1 --2- 172.21.15.154:0/4294447880 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6940a8de0 0x7fc694145de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 1 -- 172.21.15.154:0/4294447880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6940b2260 msgr2=0x7fc6941364b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 1 --2- 172.21.15.154:0/4294447880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6940b2260 0x7fc6941364b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 10 monclient: _finish_hunting 0 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 1 monclient: found mon.noname-b 2023-12-07T07:32:50.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.155+0000 7fc693fff640 1 -- 172.21.15.154:0/4294447880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c009a70 con 0x7fc6940a8a00 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.157+0000 7fc693fff640 1 --2- 172.21.15.154:0/4294447880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8a00 0x7fc694134ae0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc68400c4a0 tx=0x7fc68400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.157+0000 7fc690ff9640 10 client.?.objecter ms_handle_connect 0x7fc6940a8a00 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.157+0000 7fc690ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 1 -- 172.21.15.154:0/4294447880 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684005ce0 con 0x7fc6940a8a00 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: dump: 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.169 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: _finish_auth 0 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 1 -- 172.21.15.154:0/4294447880 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc684005e40 con 0x7fc6940a8a00 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 1 -- 172.21.15.154:0/4294447880 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684007270 con 0x7fc6940a8a00 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc690ff9640 10 monclient: dump: 2023-12-07T07:32:50.170 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc69aa2b640 5 monclient: authenticate success, global_id 4153 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc69aa2b640 10 monclient: _renew_subs 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc69aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6941455e0 con 0x7fc6940a8a00 2023-12-07T07:32:50.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.160+0000 7fc6927fc640 4 set_mon_vals no callback set 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.161+0000 7fc69aa2b640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.161+0000 7fc69aa2b640 10 monclient: _renew_subs 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.161+0000 7fc69aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.161+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6941447f0 con 0x7fc6940a8a00 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.161+0000 7fc69aa2b640 1 librados: init done 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.162+0000 7fc69aa2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.162+0000 7fc69aa2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.162+0000 7fc69aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.162+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc694145c30 con 0x7fc6940a8a00 2023-12-07T07:32:50.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.163+0000 7fc690ff9640 1 -- 172.21.15.154:0/4294447880 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc68401c440 con 0x7fc6940a8a00 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 1 -- 172.21.15.154:0/4294447880 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fc68401cce0 con 0x7fc6940a8a00 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 10 client.4153.objecter ms_dispatch 0x7fc694000b90 osd_map(8..8 src has 1..8) v4 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 20 client.4153.objecter dump_active .. 0 homeless 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 1 -- 172.21.15.154:0/4294447880 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc684021060 con 0x7fc6940a8a00 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.262+0000 7fc690ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:50.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.336+0000 7fc69aa2b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd", "id": 6}] 2023-12-07T07:32:50.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.336+0000 7fc69aa2b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd", "id": 6}] 2023-12-07T07:32:50.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.336+0000 7fc69aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.336+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd", "id": 6} v 0) v1 -- 0x7fc694067ac0 con 0x7fc6940a8a00 2023-12-07T07:32:50.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.403+0000 7fc690ff9640 1 -- 172.21.15.154:0/4294447880 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc684017070 con 0x7fc6940a8a00 2023-12-07T07:32:50.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.403+0000 7fc690ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd", "id": 6}] 2023-12-07T07:32:50.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.403+0000 7fc690ff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:50.406 INFO:teuthology.orchestra.run.smithi154.stdout:6 2023-12-07T07:32:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 10 librados: watch_flush enter 2023-12-07T07:32:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 10 librados: watch_flush exit 2023-12-07T07:32:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 20 client.4153.objecter shutdown clearing up homeless session... 2023-12-07T07:32:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 10 client.4153.objecter successfully canceled tick 2023-12-07T07:32:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 10 monclient: shutdown 2023-12-07T07:32:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8a00 msgr2=0x7fc694134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 --2- 172.21.15.154:0/4294447880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8a00 0x7fc694134ae0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc68400c4a0 tx=0x7fc68400c960 comp rx=0 tx=0).stop 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 shutdown_connections 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 --2- 172.21.15.154:0/4294447880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc6940b2260 0x7fc6941364b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 --2- 172.21.15.154:0/4294447880 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6940a8de0 0x7fc694145de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 --2- 172.21.15.154:0/4294447880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6940a8a00 0x7fc694134ae0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 >> 172.21.15.154:0/4294447880 conn(0x7fc694061a90 msgr2=0x7fc694138de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 shutdown_connections 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 -- 172.21.15.154:0/4294447880 wait complete. 2023-12-07T07:32:50.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.404+0000 7fc69aa2b640 1 librados: shutdown 2023-12-07T07:32:50.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new f35f7cd5-7ce9-4336-a36d-836f1cff51c0 7 2023-12-07T07:32:50.423 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:50.420+0000 7f141e62b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-07T07:32:50.478 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:50.476+0000 7f753c845040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-07T07:32:50.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.514+0000 7f60c651d640 10 monclient: get_monmap_and_config 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.514+0000 7f60c651d640 10 monclient: build_initial_monmap 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.514+0000 7f60c651d640 10 monclient: monmap: 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:50.515603+0000 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:50.515603+0000 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:50.517 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.515+0000 7f60c651d640 1 Processor -- start 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.515+0000 7f60c651d640 1 -- start start 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.515+0000 7f60c651d640 10 monclient: init 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60c00a8a00 addr v1:172.21.15.154:6789/0 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60c00b2260 0x7f60c00a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.519 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient(hunting): picked mon.noname-c con 0x7f60c00b2260 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60c013f080 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c00aa970 con 0x7f60c013f080 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c00aac50 con 0x7f60c00a8a00 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c01421b0 con 0x7f60c00b2260 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c651d640 10 monclient(hunting): authenticate will time out at 1264.873307s 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60bf7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60c00b2260 0x7f60c00a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c4aba640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f60c013f080 0x7f60c013f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.154:45744/0 (socket says 172.21.15.154:45744) 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.516+0000 7f60c4aba640 1 -- 172.21.15.154:0/2834024535 learned_addr learned my addr 172.21.15.154:0/2834024535 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.517+0000 7f60bf7fe640 10 monclient(hunting): get_auth_request con 0x7f60c00b2260 auth_method 0 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.517+0000 7f60bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.517+0000 7f60bf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.520 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.517+0000 7f60bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.521 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.518+0000 7f60bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.521 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.518+0000 7f60bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.521 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.518+0000 7f60bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.521 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.519+0000 7f60beffd640 1 -- 172.21.15.154:0/2834024535 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708850475 0 0) 0x7f60c00aa970 con 0x7f60c013f080 2023-12-07T07:32:50.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.519+0000 7f60beffd640 10 monclient(hunting): _negotiate 2023-12-07T07:32:50.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.519+0000 7f60beffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.519+0000 7f60beffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.519+0000 7f60beffd640 10 monclient(hunting): my global_id is 4137 2023-12-07T07:32:50.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.519+0000 7f60beffd640 1 -- 172.21.15.154:0/2834024535 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a8003520 con 0x7f60c013f080 2023-12-07T07:32:50.523 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2023-12-07T07:32:50.523 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 1 -- 172.21.15.154:0/2834024535 >> v1:172.21.15.154:6789/0 conn(0x7f60c00a8a00 legacy=0x7f60c01425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.524 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 1 -- 172.21.15.154:0/2834024535 >> v1:172.21.15.97:6789/0 conn(0x7f60c013f080 legacy=0x7f60c013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.524 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:32:50.524 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 1 monclient: found mon.noname-c 2023-12-07T07:32:50.524 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:50.524 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.524 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.521+0000 7f60bf7fe640 1 -- 172.21.15.154:0/2834024535 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c00aaf30 con 0x7f60c00b2260 2023-12-07T07:32:50.527 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.525+0000 7f60bf7fe640 1 --2- 172.21.15.154:0/2834024535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00b2260 0x7f60c00a8de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60b00097a0 tx=0x7f60b002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cc796ca73dc9bcd server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 1 -- 172.21.15.154:0/2834024535 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b002e070 con 0x7f60c00b2260 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: dump: 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.535 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.535 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.535 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: _finish_auth 0 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 1 -- 172.21.15.154:0/2834024535 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b00315a0 con 0x7f60c00b2260 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 1 -- 172.21.15.154:0/2834024535 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0031930 con 0x7f60c00b2260 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60beffd640 10 monclient: dump: 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.536 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60c651d640 5 monclient: authenticate success, global_id 4156 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60c651d640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.529+0000 7f60c651d640 4 set_mon_vals no callback set 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 10 monclient: shutdown 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 1 -- 172.21.15.154:0/2834024535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00b2260 msgr2=0x7f60c00a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 1 --2- 172.21.15.154:0/2834024535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00b2260 0x7f60c00a8de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60b00097a0 tx=0x7f60b002b840 comp rx=0 tx=0).stop 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 1 -- 172.21.15.154:0/2834024535 shutdown_connections 2023-12-07T07:32:50.537 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 1 --2- 172.21.15.154:0/2834024535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00b2260 0x7f60c00a8de0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.538 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 1 -- 172.21.15.154:0/2834024535 >> 172.21.15.154:0/2834024535 conn(0x7f60c0061a90 msgr2=0x7f60c00b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:50.538 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 1 -- 172.21.15.154:0/2834024535 shutdown_connections 2023-12-07T07:32:50.538 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.530+0000 7f60c651d640 1 -- 172.21.15.154:0/2834024535 wait complete. 2023-12-07T07:32:50.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 10 monclient: build_initial_monmap 2023-12-07T07:32:50.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 10 monclient: monmap: 2023-12-07T07:32:50.541 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2023-12-07T07:32:50.541 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 1 librados: starting msgr at 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 1 librados: starting objecter 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 1 Processor -- start 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 1 -- start start 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 1 librados: setting wanted keys 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 1 librados: calling monclient init 2023-12-07T07:32:50.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.531+0000 7f60c651d640 10 monclient: init 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00a8a00 0x7f60c0139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): picked mon.noname-b con 0x7f60c00a8a00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60c00b2260 0x7f60c0134870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): picked mon.noname-a con 0x7f60c00b2260 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f60c013f080 0x7f60c01d77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): picked mon.noname-c con 0x7f60c013f080 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c013fc40 con 0x7f60c00b2260 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c01d7030 con 0x7f60c00a8a00 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c01d7310 con 0x7f60c013f080 2023-12-07T07:32:50.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60c651d640 10 monclient(hunting): authenticate will time out at 1264.889628s 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.532+0000 7f60bffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00a8a00 0x7f60c0139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f60c013f080 0x7f60c01d77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60c00b2260 0x7f60c0134870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00a8a00 0x7f60c0139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:48522/0 (socket says 172.21.15.154:48522) 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 1 -- 172.21.15.154:0/259988001 learned_addr learned my addr 172.21.15.154:0/259988001 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 10 monclient(hunting): get_auth_request con 0x7f60c00a8a00 auth_method 0 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 10 monclient(hunting): get_auth_request con 0x7f60c013f080 auth_method 0 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bf7fe640 10 monclient(hunting): get_auth_request con 0x7f60c00b2260 auth_method 0 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bf7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.533+0000 7f60c4aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.534+0000 7f60bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.534+0000 7f60bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.534+0000 7f60bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 10 monclient(hunting): handle_auth_done global_id 4162 payload 824 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 1 -- 172.21.15.154:0/259988001 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f60c013f080 msgr2=0x7f60c01d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 1 --2- 172.21.15.154:0/259988001 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f60c013f080 0x7f60c01d77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 1 -- 172.21.15.154:0/259988001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60c00b2260 msgr2=0x7f60c0134870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 1 --2- 172.21.15.154:0/259988001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60c00b2260 0x7f60c0134870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 10 monclient: _finish_hunting 0 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 1 monclient: found mon.noname-b 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.536+0000 7f60bffff640 1 -- 172.21.15.154:0/259988001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0009400 con 0x7f60c00a8a00 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.538+0000 7f60bffff640 1 --2- 172.21.15.154:0/259988001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00a8a00 0x7f60c0139720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c450 tx=0x7f60b400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.539+0000 7f60bcff9640 10 client.?.objecter ms_handle_connect 0x7f60c00a8a00 2023-12-07T07:32:50.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.539+0000 7f60bcff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 1 -- 172.21.15.154:0/259988001 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b40052d0 con 0x7f60c00a8a00 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: dump: 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.548 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: _finish_auth 0 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: _check_auth_tickets 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 1 -- 172.21.15.154:0/259988001 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b4005430 con 0x7f60c00a8a00 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60c651d640 5 monclient: authenticate success, global_id 4162 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 1 -- 172.21.15.154:0/259988001 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4012870 con 0x7f60c00a8a00 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60bcff9640 10 monclient: dump: 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.549 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 10 monclient: _renew_subs 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0135fb0 con 0x7f60c00a8a00 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.542+0000 7f60be7fc640 4 set_mon_vals no callback set 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 10 monclient: _renew_subs 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c013b030 con 0x7f60c00a8a00 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.543+0000 7f60c651d640 1 librados: init done 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.544+0000 7f60c651d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.544+0000 7f60c651d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.544+0000 7f60c651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.544+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c0067ac0 con 0x7f60c00a8a00 2023-12-07T07:32:50.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 1 -- 172.21.15.154:0/259988001 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f60b4012cd0 con 0x7f60c00a8a00 2023-12-07T07:32:50.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 1 -- 172.21.15.154:0/259988001 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f60b4025a90 con 0x7f60c00a8a00 2023-12-07T07:32:50.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 10 client.4162.objecter ms_dispatch 0x7f60c0000b90 osd_map(9..9 src has 1..9) v4 2023-12-07T07:32:50.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2023-12-07T07:32:50.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2023-12-07T07:32:50.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 20 client.4162.objecter dump_active .. 0 homeless 2023-12-07T07:32:50.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 1 -- 172.21.15.154:0/259988001 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f60b4025ca0 con 0x7f60c00a8a00 2023-12-07T07:32:50.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.644+0000 7f60bcff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:50.696 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:50.694+0000 7f141e62b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-07T07:32:50.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.716+0000 7f60c651d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f35f7cd5-7ce9-4336-a36d-836f1cff51c0", "id": 7}] 2023-12-07T07:32:50.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.716+0000 7f60c651d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f35f7cd5-7ce9-4336-a36d-836f1cff51c0", "id": 7}] 2023-12-07T07:32:50.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.716+0000 7f60c651d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:32:50.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.716+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f35f7cd5-7ce9-4336-a36d-836f1cff51c0", "id": 7} v 0) v1 -- 0x7f60c013c690 con 0x7f60c00a8a00 2023-12-07T07:32:50.750 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:50.748+0000 7f141e62b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-07T07:32:50.761 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:50.759+0000 7f753c845040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-07T07:32:50.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.787+0000 7f60bcff9640 1 -- 172.21.15.154:0/259988001 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f35f7cd5-7ce9-4336-a36d-836f1cff51c0", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f60b40035f0 con 0x7f60c00a8a00 2023-12-07T07:32:50.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.787+0000 7f60bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f35f7cd5-7ce9-4336-a36d-836f1cff51c0", "id": 7}] 2023-12-07T07:32:50.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.787+0000 7f60bcff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:50.790 INFO:teuthology.orchestra.run.smithi154.stdout:7 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.787+0000 7f60c651d640 10 librados: watch_flush enter 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.787+0000 7f60c651d640 10 librados: watch_flush exit 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.787+0000 7f60c651d640 20 client.4162.objecter shutdown clearing up homeless session... 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.787+0000 7f60c651d640 10 client.4162.objecter successfully canceled tick 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 10 monclient: shutdown 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00a8a00 msgr2=0x7f60c0139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 --2- 172.21.15.154:0/259988001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00a8a00 0x7f60c0139720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c450 tx=0x7f60b400c910 comp rx=0 tx=0).stop 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 shutdown_connections 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 --2- 172.21.15.154:0/259988001 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f60c013f080 0x7f60c01d77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 --2- 172.21.15.154:0/259988001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60c00b2260 0x7f60c0134870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 --2- 172.21.15.154:0/259988001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c00a8a00 0x7f60c0139720 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 >> 172.21.15.154:0/259988001 conn(0x7f60c0061a90 msgr2=0x7f60c00b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:50.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 shutdown_connections 2023-12-07T07:32:50.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 -- 172.21.15.154:0/259988001 wait complete. 2023-12-07T07:32:50.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-12-07T07:32:50.788+0000 7f60c651d640 1 librados: shutdown 2023-12-07T07:32:50.801 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.801 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-07T07:32:50.801 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-12-07T07:32:50.804 INFO:tasks.ceph.osd.0:Started 2023-12-07T07:32:50.804 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.804 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-07T07:32:50.804 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-12-07T07:32:50.806 INFO:tasks.ceph.osd.1:Started 2023-12-07T07:32:50.806 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.806 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-07T07:32:50.806 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-12-07T07:32:50.811 INFO:tasks.ceph.osd.2:Started 2023-12-07T07:32:50.811 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.811 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-07T07:32:50.812 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-12-07T07:32:50.813 INFO:tasks.ceph.osd.3:Started 2023-12-07T07:32:50.814 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.814 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-07T07:32:50.814 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-12-07T07:32:50.816 INFO:tasks.ceph.osd.4:Started 2023-12-07T07:32:50.816 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.817 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-07T07:32:50.817 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-07T07:32:50.817 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:50.814+0000 7f753c845040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-07T07:32:50.819 INFO:tasks.ceph.osd.5:Started 2023-12-07T07:32:50.819 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.819 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-07T07:32:50.819 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-12-07T07:32:50.821 INFO:tasks.ceph.osd.6:Started 2023-12-07T07:32:50.821 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-07T07:32:50.821 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-07T07:32:50.821 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-07T07:32:50.824 INFO:tasks.ceph.osd.7:Started 2023-12-07T07:32:50.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T07:32:50.873 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:50.871+0000 7f141e62b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-07T07:32:50.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.943+0000 7f73d0c2b640 10 monclient: get_monmap_and_config 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.943+0000 7f73d0c2b640 10 monclient: build_initial_monmap 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.943+0000 7f73d0c2b640 10 monclient: monmap: 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:50.944455+0000 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:50.944455+0000 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:50.947 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:50.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 1 Processor -- start 2023-12-07T07:32:50.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 1 -- start start 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient: init 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8a30 0x7f73cc1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f73cc0a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73cc0a8e10 0x7f73cc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient(hunting): picked mon.noname-a con 0x7f73cc0a8e10 addr v2:172.21.15.97:3300/0 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73cc1454a0 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc0aac10 con 0x7f73cc1454a0 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc0aaa10 con 0x7f73cc0a8e10 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0aaef0 con 0x7f73cc0a8a30 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.944+0000 7f73d0c2b640 10 monclient(hunting): authenticate will time out at 1293.323251s 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.945+0000 7f73ca59c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8a30 0x7f73cc1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.945+0000 7f73c3fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f73cc0a8e10 0x7f73cc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73cad9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f73cc1454a0 0x7f73cc145930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37644/0 (socket says 172.21.15.97:37644) 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73cad9d640 1 -- 172.21.15.97:0/3928099872 learned_addr learned my addr 172.21.15.97:0/3928099872 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73ca59c640 10 monclient(hunting): get_auth_request con 0x7f73cc0a8a30 auth_method 0 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73ca59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73ca59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73c3fff640 10 monclient(hunting): get_auth_request con 0x7f73cc0a8e10 auth_method 0 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73c3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.946+0000 7f73ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.947+0000 7f73c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.947+0000 7f73c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.947+0000 7f73c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 10 monclient(hunting): handle_auth_done global_id 4148 payload 274 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 1 -- 172.21.15.97:0/3928099872 >> v2:172.21.15.97:3300/0 conn(0x7f73cc0a8e10 msgr2=0x7f73cc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 1 --2- 172.21.15.97:0/3928099872 >> v2:172.21.15.97:3300/0 conn(0x7f73cc0a8e10 0x7f73cc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 1 -- 172.21.15.97:0/3928099872 >> v1:172.21.15.97:6789/0 conn(0x7f73cc1454a0 legacy=0x7f73cc145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 10 monclient: _finish_hunting 0 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 1 monclient: found mon.noname-b 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:50.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:32:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.949+0000 7f73ca59c640 1 -- 172.21.15.97:0/3928099872 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc0a97f0 con 0x7f73cc0a8a30 2023-12-07T07:32:50.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.952+0000 7f73ca59c640 1 --2- 172.21.15.97:0/3928099872 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8a30 0x7f73cc1424e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0097d0 tx=0x7f73bc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17adb0b239a0c50a server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:50.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 1 -- 172.21.15.97:0/3928099872 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc02d040 con 0x7f73cc0a8a30 2023-12-07T07:32:50.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:32:50.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: dump: 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: _finish_auth 0 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: _check_auth_tickets 2023-12-07T07:32:50.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 1 -- 172.21.15.97:0/3928099872 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc031810 con 0x7f73cc0a8a30 2023-12-07T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73d0c2b640 5 monclient: authenticate success, global_id 4148 2023-12-07T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73d0c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:50.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73d0c2b640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 1 -- 172.21.15.97:0/3928099872 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc031ba0 con 0x7f73cc0a8a30 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73d0c2b640 4 set_mon_vals no callback set 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.957+0000 7f73c9d9b640 10 monclient: dump: 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.965 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.966 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.966 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.966 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:50.966 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.966 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.958+0000 7f73d0c2b640 10 monclient: shutdown 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.958+0000 7f73d0c2b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.958+0000 7f73d0c2b640 1 -- 172.21.15.97:0/3928099872 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8a30 msgr2=0x7f73cc1424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.958+0000 7f73d0c2b640 1 --2- 172.21.15.97:0/3928099872 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8a30 0x7f73cc1424e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0097d0 tx=0x7f73bc02b8b0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.958+0000 7f73d0c2b640 1 -- 172.21.15.97:0/3928099872 shutdown_connections 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.959+0000 7f73d0c2b640 1 --2- 172.21.15.97:0/3928099872 >> v2:172.21.15.97:3300/0 conn(0x7f73cc0a8e10 0x7f73cc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.959+0000 7f73d0c2b640 1 --2- 172.21.15.97:0/3928099872 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8a30 0x7f73cc1424e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.959+0000 7f73d0c2b640 1 -- 172.21.15.97:0/3928099872 >> 172.21.15.97:0/3928099872 conn(0x7f73cc061a30 msgr2=0x7f73cc0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.959+0000 7f73d0c2b640 1 -- 172.21.15.97:0/3928099872 shutdown_connections 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.959+0000 7f73d0c2b640 1 -- 172.21.15.97:0/3928099872 wait complete. 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.960+0000 7f73d0c2b640 10 monclient: build_initial_monmap 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.960+0000 7f73d0c2b640 10 monclient: monmap: 2023-12-07T07:32:50.967 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:50.968 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.960+0000 7f73d0c2b640 1 librados: starting msgr at 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.960+0000 7f73d0c2b640 1 librados: starting objecter 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.961+0000 7f73d0c2b640 1 Processor -- start 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.961+0000 7f73d0c2b640 1 -- start start 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.961+0000 7f73d0c2b640 1 librados: setting wanted keys 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.961+0000 7f73d0c2b640 1 librados: calling monclient init 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.961+0000 7f73d0c2b640 10 monclient: init 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73cc0a8a30 0x7f73cc1d65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f73cc0a8a30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:50.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8e10 0x7f73cc1d6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): picked mon.noname-c con 0x7f73cc0a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc1454a0 0x7f73cc1d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): picked mon.noname-a con 0x7f73cc1454a0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0b6070 con 0x7f73cc1454a0 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc1d6120 con 0x7f73cc0a8a30 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc137710 con 0x7f73cc0a8e10 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73d0c2b640 10 monclient(hunting): authenticate will time out at 1293.340780s 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73c3fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8e10 0x7f73cc1d6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.962+0000 7f73cad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc1454a0 0x7f73cc1d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73cad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc1454a0 0x7f73cc1d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33800/0 (socket says 172.21.15.97:33800) 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73cad9d640 1 -- 172.21.15.97:0/134752345 learned_addr learned my addr 172.21.15.97:0/134752345 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73cad9d640 10 monclient(hunting): get_auth_request con 0x7f73cc1454a0 auth_method 0 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73cad9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73cad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73c3fff640 10 monclient(hunting): get_auth_request con 0x7f73cc0a8e10 auth_method 0 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73c3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.963+0000 7f73c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.964+0000 7f73cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.964+0000 7f73cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.964+0000 7f73cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.964+0000 7f73c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.964+0000 7f73c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.964+0000 7f73c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.965+0000 7f73ca59c640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73cc0a8a30 0x7f73cc1d65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.966+0000 7f73ca59c640 10 monclient(hunting): get_auth_request con 0x7f73cc0a8a30 auth_method 0 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.966+0000 7f73ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.966+0000 7f73ca59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.966+0000 7f73ca59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.967+0000 7f73ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.967+0000 7f73ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.967+0000 7f73ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 10 monclient(hunting): handle_auth_done global_id 4151 payload 824 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 1 -- 172.21.15.97:0/134752345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73cc0a8a30 msgr2=0x7f73cc1d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73cc0a8a30 0x7f73cc1d65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 1 -- 172.21.15.97:0/134752345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc1454a0 msgr2=0x7f73cc1d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc1454a0 0x7f73cc1d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 10 monclient: _finish_hunting 0 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 1 monclient: found mon.noname-c 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:50.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:32:50.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.968+0000 7f73c3fff640 1 -- 172.21.15.97:0/134752345 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc009400 con 0x7f73cc0a8e10 2023-12-07T07:32:50.977 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-12-07T07:32:50.969+0000 7f753c845040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-07T07:32:50.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.971+0000 7f73c2ffd640 10 client.?.objecter ms_handle_connect 0x7f73cc0a8e10 2023-12-07T07:32:50.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.971+0000 7f73c2ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:50.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.971+0000 7f73c3fff640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8e10 0x7f73cc1d6af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f73b400e850 tx=0x7f73b400ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b40095b0 con 0x7f73cc0a8e10 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73c2ffd640 10 monclient: dump: 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.983 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.984 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.984 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:50.984 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73c2ffd640 10 monclient: _finish_auth 0 2023-12-07T07:32:50.984 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73c2ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:50.984 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:50.984 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 5 monclient: authenticate success, global_id 4151 2023-12-07T07:32:50.984 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 10 monclient: _renew_subs 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc1d2b60 con 0x7f73cc0a8e10 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 10 monclient: _renew_subs 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc1d37c0 con 0x7f73cc0a8e10 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.977+0000 7f73d0c2b640 1 librados: init done 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.978+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b400c940 con 0x7f73cc0a8e10 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.978+0000 7f73c2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.978+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b40095b0 con 0x7f73cc0a8e10 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.978+0000 7f73c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:50.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.978+0000 7f73c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.978+0000 7f73c959a640 4 set_mon_vals no callback set 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.978+0000 7f73c2ffd640 10 monclient: dump: 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.979+0000 7f73d0c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.986 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.979+0000 7f73d0c2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:50.987 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.979+0000 7f73d0c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:50.987 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:50.979+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b0005040 con 0x7f73cc0a8e10 2023-12-07T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.028+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f73b4005990 con 0x7f73cc0a8e10 2023-12-07T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.029+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f73b400cdb0 con 0x7f73cc0a8e10 2023-12-07T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.029+0000 7f73c2ffd640 10 client.4151.objecter ms_dispatch 0x7f73cc000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.029+0000 7f73c2ffd640 3 client.4151.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.029+0000 7f73c2ffd640 3 client.4151.objecter handle_osd_map decoding full epoch 10 2023-12-07T07:32:51.031 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.029+0000 7f73c2ffd640 20 client.4151.objecter dump_active .. 0 homeless 2023-12-07T07:32:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.207+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f73b4021060 con 0x7f73cc0a8e10 2023-12-07T07:32:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.207+0000 7f73c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.207+0000 7f73c2ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.330+0000 7f73d0c2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:32:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.330+0000 7f73d0c2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:32:51.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.330+0000 7f73d0c2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:32:51.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.330+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73b00055c0 con 0x7f73cc0a8e10 2023-12-07T07:32:51.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.598+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7f73b4005990 con 0x7f73cc0a8e10 2023-12-07T07:32:51.718 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:32:51.718 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":10,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:32:50.728917+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:32:51.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.716+0000 7f73c2ffd640 1 -- 172.21.15.97:0/134752345 <== mon.2 v2:172.21.15.97:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f73b4011350 con 0x7f73cc0a8e10 2023-12-07T07:32:51.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.716+0000 7f73c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.716+0000 7f73c2ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 10 librados: watch_flush enter 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 10 librados: watch_flush exit 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 20 client.4151.objecter shutdown clearing up homeless session... 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 10 client.4151.objecter successfully canceled tick 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 10 monclient: shutdown 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8e10 msgr2=0x7f73cc1d6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8e10 0x7f73cc1d6af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f73b400e850 tx=0x7f73b400ed10 comp rx=0 tx=0).stop 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 shutdown_connections 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73cc1454a0 0x7f73cc1d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73cc0a8e10 0x7f73cc1d6af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 --2- 172.21.15.97:0/134752345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73cc0a8a30 0x7f73cc1d65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 >> 172.21.15.97:0/134752345 conn(0x7f73cc061a30 msgr2=0x7f73cc13d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:51.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 shutdown_connections 2023-12-07T07:32:51.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 -- 172.21.15.97:0/134752345 wait complete. 2023-12-07T07:32:51.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:32:51.717+0000 7f73d0c2b640 1 librados: shutdown 2023-12-07T07:32:51.735 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-07T07:32:51.736 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-07T07:32:52.541 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T07:32:52.682 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.680+0000 7f2ceca90640 10 monclient: get_monmap_and_config 2023-12-07T07:32:52.682 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.680+0000 7f2ceca90640 10 monclient: build_initial_monmap 2023-12-07T07:32:52.682 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.680+0000 7f2ceca90640 10 monclient: monmap: 2023-12-07T07:32:52.682 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:52.681104+0000 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:52.681104+0000 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:32:52.683 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.680+0000 7f2ceca90640 1 Processor -- start 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.680+0000 7f2ceca90640 1 -- start start 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.680+0000 7f2ceca90640 10 monclient: init 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2ce80a8a00 0x7f2ce80a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce80a8a00 addr v2:172.21.15.97:3300/0 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ce8145a40 addr v1:172.21.15.97:6789/0 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2ce8145e80 0x7f2ce8142630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:52.685 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce8145e80 addr v2:172.21.15.154:3300/0 2023-12-07T07:32:52.686 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce81455b0 con 0x7f2ce8145a40 2023-12-07T07:32:52.686 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:52.686 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80aa970 con 0x7f2ce80a8a00 2023-12-07T07:32:52.686 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:52.686 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80aac50 con 0x7f2ce8145e80 2023-12-07T07:32:52.686 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:52.686 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.682+0000 7f2ceca90640 10 monclient(hunting): authenticate will time out at 1295.060446s 2023-12-07T07:32:52.814 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.810+0000 7f2ce659c640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2ce80a8a00 0x7f2ce80a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:52.814 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.811+0000 7f2ce659c640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2ce80a8a00 0x7f2ce80a8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33884/0 (socket says 172.21.15.97:33884) 2023-12-07T07:32:52.814 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.811+0000 7f2ce659c640 1 -- 172.21.15.97:0/1459337562 learned_addr learned my addr 172.21.15.97:0/1459337562 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:32:52.814 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.811+0000 7f2ce659c640 10 monclient(hunting): get_auth_request con 0x7f2ce80a8a00 auth_method 0 2023-12-07T07:32:52.814 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.811+0000 7f2ce659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:52.814 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.811+0000 7f2ce659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:52.814 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.811+0000 7f2ce659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:52.815 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.812+0000 7f2ce659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:52.815 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.812+0000 7f2ce659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:52.815 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.812+0000 7f2ce659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:52.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 10 monclient(hunting): handle_auth_done global_id 4155 payload 274 2023-12-07T07:32:52.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 1 -- 172.21.15.97:0/1459337562 >> v2:172.21.15.154:3300/0 conn(0x7f2ce8145e80 msgr2=0x7f2ce8142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:52.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 1 --2- 172.21.15.97:0/1459337562 >> v2:172.21.15.154:3300/0 conn(0x7f2ce8145e80 0x7f2ce8142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:52.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 1 -- 172.21.15.97:0/1459337562 >> v1:172.21.15.97:6789/0 conn(0x7f2ce8145a40 legacy=0x7f2ce8141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:32:52.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 10 monclient: _finish_hunting 0 2023-12-07T07:32:52.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 1 monclient: found mon.noname-a 2023-12-07T07:32:52.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:52.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:32:52.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.822+0000 7f2ce659c640 1 -- 172.21.15.97:0/1459337562 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80aaf30 con 0x7f2ce80a8a00 2023-12-07T07:32:52.857 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.855+0000 7f2ce659c640 1 --2- 172.21.15.97:0/1459337562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80a8de0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80b3d80 tx=0x7f2cd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9164b4390f57a428 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:52.972 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ce559a640 1 -- 172.21.15.97:0/1459337562 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8031870 con 0x7f2ce80a8a00 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ce559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ce559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ce559a640 10 monclient: dump: 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:52.973 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:32:52.974 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ce559a640 10 monclient: _finish_auth 0 2023-12-07T07:32:52.974 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ce559a640 10 monclient: _check_auth_tickets 2023-12-07T07:32:52.974 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ce559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:52.974 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ceca90640 5 monclient: authenticate success, global_id 4155 2023-12-07T07:32:52.975 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.970+0000 7f2ceca90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:32:52.998 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ce559a640 1 -- 172.21.15.97:0/1459337562 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd80319d0 con 0x7f2ce80a8a00 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ce559a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ce559a640 1 -- 172.21.15.97:0/1459337562 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8031d60 con 0x7f2ce80a8a00 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ce559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ce559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ce559a640 10 monclient: dump: 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:52.999 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:32:53.000 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ceca90640 10 monclient: get_monmap_and_config success 2023-12-07T07:32:53.000 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.996+0000 7f2ceca90640 4 set_mon_vals no callback set 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 10 monclient: shutdown 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 1 -- 172.21.15.97:0/1459337562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 msgr2=0x7f2ce80a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 1 --2- 172.21.15.97:0/1459337562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80a8de0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80b3d80 tx=0x7f2cd802b910 comp rx=0 tx=0).stop 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 1 -- 172.21.15.97:0/1459337562 shutdown_connections 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 1 --2- 172.21.15.97:0/1459337562 >> v2:172.21.15.154:3300/0 conn(0x7f2ce8145e80 0x7f2ce8142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 1 --2- 172.21.15.97:0/1459337562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80a8de0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.997+0000 7f2ceca90640 1 -- 172.21.15.97:0/1459337562 >> 172.21.15.97:0/1459337562 conn(0x7f2ce8066e80 msgr2=0x7f2ce80b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.998+0000 7f2ceca90640 1 -- 172.21.15.97:0/1459337562 shutdown_connections 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.998+0000 7f2ceca90640 1 -- 172.21.15.97:0/1459337562 wait complete. 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.999+0000 7f2ceca90640 10 monclient: build_initial_monmap 2023-12-07T07:32:53.001 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:52.999+0000 7f2ceca90640 10 monclient: monmap: 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:32:53.002 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:32:53.005 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.000+0000 7f2ceca90640 1 librados: starting msgr at 2023-12-07T07:32:53.005 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.000+0000 7f2ceca90640 1 librados: starting objecter 2023-12-07T07:32:53.005 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.000+0000 7f2ceca90640 1 Processor -- start 2023-12-07T07:32:53.005 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.000+0000 7f2ceca90640 1 -- start start 2023-12-07T07:32:53.005 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.000+0000 7f2ceca90640 1 librados: setting wanted keys 2023-12-07T07:32:53.005 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.000+0000 7f2ceca90640 1 librados: calling monclient init 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.000+0000 7f2ceca90640 10 monclient: init 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient: _reopen_session rank -1 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce80a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2ce8145a40 0x7f2ce80add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce8145a40 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ce8145e80 0x7f2ce81d71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce8145e80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce81d6d20 con 0x7f2ce80a8a00 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8145290 con 0x7f2ce8145e80 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:32:53.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80aea10 con 0x7f2ce8145a40 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.001+0000 7f2ceca90640 10 monclient(hunting): _renew_subs 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ceca90640 10 monclient(hunting): authenticate will time out at 1295.380315s 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ce659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ce659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33892/0 (socket says 172.21.15.97:33892) 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ce659c640 1 -- 172.21.15.97:0/842338541 learned_addr learned my addr 172.21.15.97:0/842338541 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ce659c640 10 monclient(hunting): get_auth_request con 0x7f2ce80a8a00 auth_method 0 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ce659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ce659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.002+0000 7f2ce659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.003+0000 7f2ce659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.003+0000 7f2ce659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:32:53.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.003+0000 7f2ce659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:32:53.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 10 monclient(hunting): handle_auth_done global_id 4158 payload 824 2023-12-07T07:32:53.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 1 -- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2ce8145a40 msgr2=0x7f2ce80add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:53.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2ce8145a40 0x7f2ce80add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:53.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 1 -- 172.21.15.97:0/842338541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ce8145e80 msgr2=0x7f2ce81d71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:53.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ce8145e80 0x7f2ce81d71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:53.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 10 monclient: _finish_hunting 0 2023-12-07T07:32:53.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 1 monclient: found mon.noname-a 2023-12-07T07:32:53.011 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:32:53.011 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:32:53.011 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.007+0000 7f2ce659c640 1 -- 172.21.15.97:0/842338541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80094d0 con 0x7f2ce80a8a00 2023-12-07T07:32:53.013 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.011+0000 7f2ce659c640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80ad7e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80090b0 tx=0x7f2cd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:53.013 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.011+0000 7f2cd6ffd640 10 client.?.objecter ms_handle_connect 0x7f2ce80a8a00 2023-12-07T07:32:53.013 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.011+0000 7f2cd6ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:32:53.018 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 1 -- 172.21.15.97:0/842338541 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80307d0 con 0x7f2ce80a8a00 2023-12-07T07:32:53.018 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:53.018 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: dump: 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: _finish_auth 0 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:32:53.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 1 -- 172.21.15.97:0/842338541 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8030930 con 0x7f2ce80a8a00 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 1 -- 172.21.15.97:0/842338541 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8030ce0 con 0x7f2ce80a8a00 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2cd6ffd640 10 monclient: dump: 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:32:53.020 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:32:53.021 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2ce4d99640 4 set_mon_vals no callback set 2023-12-07T07:32:53.021 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2ceca90640 5 monclient: authenticate success, global_id 4158 2023-12-07T07:32:53.021 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2ceca90640 10 monclient: _renew_subs 2023-12-07T07:32:53.021 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.016+0000 7f2ceca90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.017+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce80b0060 con 0x7f2ce80a8a00 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.017+0000 7f2ceca90640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.017+0000 7f2ceca90640 10 monclient: _renew_subs 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.017+0000 7f2ceca90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.017+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce81d78d0 con 0x7f2ce80a8a00 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.017+0000 7f2ceca90640 1 librados: init done 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.018+0000 7f2ceca90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.018+0000 7f2ceca90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.018+0000 7f2ceca90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:53.022 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.018+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce80b01f0 con 0x7f2ce80a8a00 2023-12-07T07:32:53.028 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:53.026+0000 7f13bf13a640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-07T07:32:53.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.051+0000 7f2cd6ffd640 1 -- 172.21.15.97:0/842338541 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 3) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f2cd804d450 con 0x7f2ce80a8a00 2023-12-07T07:32:53.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.052+0000 7f2cd6ffd640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2cc806c670 0x7f2cc806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:32:53.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.052+0000 7f2ce5d9b640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2cc806c670 0x7f2cc806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:32:53.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.052+0000 7f2ce5d9b640 10 monclient: get_auth_request con 0x7f2cc806c670 auth_method 0 2023-12-07T07:32:53.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.053+0000 7f2ce5d9b640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2cc806c670 0x7f2cc806eb30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ce813f740 tx=0x7f2cdc006ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:32:53.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.053+0000 7f2cd6ffd640 10 client.4158.objecter ms_handle_connect 0x7f2cc806c670 2023-12-07T07:32:53.152 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.150+0000 7f2cd6ffd640 1 -- 172.21.15.97:0/842338541 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2cd802d040 con 0x7f2ce80a8a00 2023-12-07T07:32:53.152 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.150+0000 7f2cd6ffd640 10 client.4158.objecter ms_dispatch 0x7f2ce8000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T07:32:53.152 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.150+0000 7f2cd6ffd640 3 client.4158.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T07:32:53.152 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.150+0000 7f2cd6ffd640 3 client.4158.objecter handle_osd_map decoding full epoch 10 2023-12-07T07:32:53.152 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.150+0000 7f2cd6ffd640 20 client.4158.objecter dump_active .. 0 homeless 2023-12-07T07:32:53.245 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T07:32:53.243+0000 7f13bf13a640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-07T07:32:53.499 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.496+0000 7f2cd6ffd640 1 -- 172.21.15.97:0/842338541 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cd8077d00 con 0x7f2ce80a8a00 2023-12-07T07:32:53.499 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.496+0000 7f2cd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:32:53.499 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.496+0000 7f2cd6ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:32:53.858 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.855+0000 7f2ceca90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:32:53.858 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.855+0000 7f2ceca90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:32:53.858 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.855+0000 7f2ceca90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:32:53.858 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:53.855+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ce80ae260 con 0x7f2ce80a8a00 2023-12-07T07:32:54.111 INFO:teuthology.misc.health.smithi097.stdout: 2023-12-07T07:32:54.111 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:32:50.728917+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:32:54.112 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.109+0000 7f2cd6ffd640 1 -- 172.21.15.97:0/842338541 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2cd807b320 con 0x7f2ce80a8a00 2023-12-07T07:32:54.112 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.109+0000 7f2cd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:32:54.112 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.109+0000 7f2cd6ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 10 librados: watch_flush enter 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 10 librados: watch_flush exit 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 20 client.4158.objecter shutdown clearing up homeless session... 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 10 client.4158.objecter successfully canceled tick 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2cc806c670 msgr2=0x7f2cc806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2cc806c670 0x7f2cc806eb30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ce813f740 tx=0x7f2cdc006ba0 comp rx=0 tx=0).stop 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 10 monclient: shutdown 2023-12-07T07:32:54.114 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 msgr2=0x7f2ce80ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.111+0000 7f2ceca90640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80ad7e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80090b0 tx=0x7f2cd80046c0 comp rx=0 tx=0).stop 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 shutdown_connections 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2cc806c670 0x7f2cc806eb30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ce8145e80 0x7f2ce81d71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2ce8145a40 0x7f2ce80add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 --2- 172.21.15.97:0/842338541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce80a8a00 0x7f2ce80ad7e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 >> 172.21.15.97:0/842338541 conn(0x7f2ce8066e80 msgr2=0x7f2ce80b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 shutdown_connections 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 -- 172.21.15.97:0/842338541 wait complete. 2023-12-07T07:32:54.115 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:32:54.112+0000 7f2ceca90640 1 librados: shutdown 2023-12-07T07:32:54.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T07:32:58.787 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:07.903 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:58.867 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:07.983 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:58.946 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:08.062 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.015 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:08.131 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.063 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:08.158 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.122 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:08.217 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.194 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:08.290 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.251 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:08.347 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.313 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:08.420 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.373 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:08.480 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.405 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-12-07T07:32:59.394+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:32:59.436 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:08.547 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.496 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:08.608 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.538 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:08.666 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.592 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-12-07T07:32:59.584+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:32:59.604 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:08.731 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.694 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:32:59.686+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:32:59.797 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-12-07T07:32:59.790+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:32:59.801 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:08.907 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.811 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:08.927 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.863 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:08.969 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:32:59.918 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:32:59.910+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:33:00.031 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:09.147 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.062 INFO:tasks.ceph.osd.3.smithi097.stderr:2023-12-07T07:33:00.054+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:33:00.121 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:09.216 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.164 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:09.281 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.175 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-12-07T07:33:00.167+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:33:00.260 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:09.356 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.331 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:09.438 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.382 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:09.498 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.436 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-12-07T07:33:00.428+0000 5dc2340 -1 Falling back to public interface 2023-12-07T07:33:00.478 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:09.573 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.504 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:09.616 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.526 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:09.643 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.614 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:09.710 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.623 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:09.751 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.688 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:09.795 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.721 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:09.817 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.746 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:09.863 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.814 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:09.930 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.832 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:09.927 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.864 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:09.975 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.876 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:09.982 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:00.936 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:00.978 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.106 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.010 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:10.106 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.024+0000 7f25ff5f5640 10 monclient: get_monmap_and_config 2023-12-07T07:33:01.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.024+0000 7f25ff5f5640 10 monclient: build_initial_monmap 2023-12-07T07:33:01.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.024+0000 7f25ff5f5640 10 monclient: monmap: 2023-12-07T07:33:01.026 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:01.026 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:33:01.025334+0000 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:33:01.025334+0000 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:01.027 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:01.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.024+0000 7f25ff5f5640 1 Processor -- start 2023-12-07T07:33:01.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 1 -- start start 2023-12-07T07:33:01.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient: init 2023-12-07T07:33:01.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25f80a8a00 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f8146150 0x7f25f80a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient(hunting): picked mon.noname-b con 0x7f25f8146150 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25f813eca0 0x7f25f813f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient(hunting): picked mon.noname-a con 0x7f25f813eca0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f81455b0 con 0x7f25f80a8a00 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f80aa970 con 0x7f25f813eca0 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80aac50 con 0x7f25f8146150 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.025+0000 7f25ff5f5640 10 monclient(hunting): authenticate will time out at 1303.403918s 2023-12-07T07:33:01.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fcb90640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f8146150 0x7f25f80a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fdb92640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25f813eca0 0x7f25f813f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fdb92640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f25f813eca0 0x7f25f813f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:33040/0 (socket says 172.21.15.97:33040) 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fdb92640 1 -- 172.21.15.97:0/3067362988 learned_addr learned my addr 172.21.15.97:0/3067362988 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fdb92640 10 monclient(hunting): get_auth_request con 0x7f25f813eca0 auth_method 0 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fdb92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fdb92640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fdb92640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fcb90640 10 monclient(hunting): get_auth_request con 0x7f25f8146150 auth_method 0 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fcb90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fcb90640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.026+0000 7f25fcb90640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.027+0000 7f25fdb92640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.027+0000 7f25fdb92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.027+0000 7f25fdb92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.027+0000 7f25fcb90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:01.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.027+0000 7f25fcb90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:01.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.027+0000 7f25fcb90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:01.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2023-12-07T07:33:01.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 1 -- 172.21.15.97:0/3067362988 >> v2:172.21.15.97:3300/0 conn(0x7f25f813eca0 msgr2=0x7f25f813f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:01.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 1 --2- 172.21.15.97:0/3067362988 >> v2:172.21.15.97:3300/0 conn(0x7f25f813eca0 0x7f25f813f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 1 -- 172.21.15.97:0/3067362988 >> v1:172.21.15.97:6789/0 conn(0x7f25f80a8a00 legacy=0x7f25f8145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:01.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 10 monclient: _finish_hunting 0 2023-12-07T07:33:01.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 1 monclient: found mon.noname-b 2023-12-07T07:33:01.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:01.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:01.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.028+0000 7f25fcb90640 1 -- 172.21.15.97:0/3067362988 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f80aaf30 con 0x7f25f8146150 2023-12-07T07:33:01.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.030+0000 7f25fcb90640 1 --2- 172.21.15.97:0/3067362988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f8146150 0x7f25f80a8de0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f25e80097a0 tx=0x7f25e802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e823d853739743 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:01.036 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 1 -- 172.21.15.97:0/3067362988 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e802e070 con 0x7f25f8146150 2023-12-07T07:33:01.036 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:01.036 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: dump: 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:01.037 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: _finish_auth 0 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 1 -- 172.21.15.97:0/3067362988 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e80315a0 con 0x7f25f8146150 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 5 monclient: authenticate success, global_id 4184 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 4 set_mon_vals no callback set 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 1 -- 172.21.15.97:0/3067362988 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8031930 con 0x7f25f8146150 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:01.038 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25effff640 10 monclient: dump: 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:01.039 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:01.042 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 10 monclient: shutdown 2023-12-07T07:33:01.042 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:01.042 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 -- 172.21.15.97:0/3067362988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f8146150 msgr2=0x7f25f80a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/3067362988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f8146150 0x7f25f80a8de0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f25e80097a0 tx=0x7f25e802b840 comp rx=0 tx=0).stop 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 -- 172.21.15.97:0/3067362988 shutdown_connections 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/3067362988 >> v2:172.21.15.97:3300/0 conn(0x7f25f813eca0 0x7f25f813f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/3067362988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f8146150 0x7f25f80a8de0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 -- 172.21.15.97:0/3067362988 >> 172.21.15.97:0/3067362988 conn(0x7f25f8066e80 msgr2=0x7f25f80b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 -- 172.21.15.97:0/3067362988 shutdown_connections 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.034+0000 7f25ff5f5640 1 -- 172.21.15.97:0/3067362988 wait complete. 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.035+0000 7f25ff5f5640 10 monclient: build_initial_monmap 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.035+0000 7f25ff5f5640 10 monclient: monmap: 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:01.043 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:01.044 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.035+0000 7f25ff5f5640 1 librados: starting msgr at 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.035+0000 7f25ff5f5640 1 librados: starting objecter 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 Processor -- start 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 -- start start 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 librados: setting wanted keys 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 librados: calling monclient init 2023-12-07T07:33:01.045 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient: init 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f80a8a00 0x7f25f8138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): picked mon.noname-c con 0x7f25f80a8a00 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25f813eca0 0x7f25f8138f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): picked mon.noname-b con 0x7f25f813eca0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25f8146150 0x7f25f81dad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): picked mon.noname-a con 0x7f25f8146150 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f81394a0 con 0x7f25f8146150 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f81dba90 con 0x7f25f813eca0 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:01.046 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80ae2b0 con 0x7f25f80a8a00 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.036+0000 7f25ff5f5640 10 monclient(hunting): authenticate will time out at 1303.414962s 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fd391640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f80a8a00 0x7f25f8138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fdb92640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25f8146150 0x7f25f81dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fcb90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25f813eca0 0x7f25f8138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fd391640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f80a8a00 0x7f25f8138a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57644/0 (socket says 172.21.15.97:57644) 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fd391640 1 -- 172.21.15.97:0/1176010556 learned_addr learned my addr 172.21.15.97:0/1176010556 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fd391640 10 monclient(hunting): get_auth_request con 0x7f25f80a8a00 auth_method 0 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fd391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fd391640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fd391640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fdb92640 10 monclient(hunting): get_auth_request con 0x7f25f8146150 auth_method 0 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fdb92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fdb92640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:01.047 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fdb92640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fcb90640 10 monclient(hunting): get_auth_request con 0x7f25f813eca0 auth_method 0 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fcb90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fcb90640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.037+0000 7f25fcb90640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fd391640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fd391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fd391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fcb90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fcb90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fcb90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fdb92640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fdb92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:01.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.038+0000 7f25fdb92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:01.049 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:10.157 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 10 monclient(hunting): handle_auth_done global_id 4210 payload 824 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 1 -- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f80a8a00 msgr2=0x7f25f8138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f80a8a00 0x7f25f8138a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 1 -- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25f8146150 msgr2=0x7f25f81dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25f8146150 0x7f25f81dad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 10 monclient: _finish_hunting 0 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 1 monclient: found mon.noname-b 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.041+0000 7f25fcb90640 1 -- 172.21.15.97:0/1176010556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e8009400 con 0x7f25f813eca0 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.043+0000 7f25fcb90640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25f813eca0 0x7f25f8138f60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f25f80aad90 tx=0x7f25e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:01.051 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.043+0000 7f25edffb640 10 client.?.objecter ms_handle_connect 0x7f25f813eca0 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.043+0000 7f25edffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 1 -- 172.21.15.97:0/1176010556 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8032840 con 0x7f25f813eca0 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: dump: 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:01.052 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 1 -- 172.21.15.97:0/1176010556 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e80329a0 con 0x7f25f813eca0 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 5 monclient: authenticate success, global_id 4210 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 10 monclient: _renew_subs 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f80b2470 con 0x7f25f813eca0 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 1 -- 172.21.15.97:0/1176010556 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8032d50 con 0x7f25f813eca0 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25edffb640 10 monclient: dump: 2023-12-07T07:33:01.053 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 10 monclient: _renew_subs 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ef7fe640 4 set_mon_vals no callback set 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f81db2a0 con 0x7f25f813eca0 2023-12-07T07:33:01.054 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.047+0000 7f25ff5f5640 1 librados: init done 2023-12-07T07:33:01.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.048+0000 7f25ff5f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:01.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.048+0000 7f25ff5f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:01.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.048+0000 7f25ff5f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:01.055 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.048+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f8144730 con 0x7f25f813eca0 2023-12-07T07:33:01.063 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:10.170 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.106 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:10.223 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.132 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:10.227 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.235 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:10.347 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.242 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:10.348 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.305 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:10.401 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.316 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.314+0000 7f25edffb640 1 -- 172.21.15.97:0/1176010556 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f25e804d420 con 0x7f25f813eca0 2023-12-07T07:33:01.317 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.314+0000 7f25edffb640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f25dc06c740 0x7f25dc06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:01.317 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.315+0000 7f25fd391640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f25dc06c740 0x7f25dc06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:01.317 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.315+0000 7f25fd391640 10 monclient: get_auth_request con 0x7f25dc06c740 auth_method 0 2023-12-07T07:33:01.317 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.315+0000 7f25edffb640 10 client.4210.objecter ms_handle_connect 0x7f25dc06c740 2023-12-07T07:33:01.317 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.315+0000 7f25fd391640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f25dc06c740 0x7f25dc06ec00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f25f4005020 tx=0x7f25f4007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:01.321 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.319+0000 7f25edffb640 1 -- 172.21.15.97:0/1176010556 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f25e807a170 con 0x7f25f813eca0 2023-12-07T07:33:01.321 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.319+0000 7f25edffb640 10 client.4210.objecter ms_dispatch 0x7f25f8000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T07:33:01.321 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.319+0000 7f25edffb640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T07:33:01.321 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.319+0000 7f25edffb640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2023-12-07T07:33:01.321 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.319+0000 7f25edffb640 20 client.4210.objecter dump_active .. 0 homeless 2023-12-07T07:33:01.332 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:10.448 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.340 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.338+0000 7f25edffb640 1 -- 172.21.15.97:0/1176010556 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25e80764b0 con 0x7f25f813eca0 2023-12-07T07:33:01.340 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.338+0000 7f25edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:01.340 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.338+0000 7f25edffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:01.353 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:10.460 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.356 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:10.463 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.365 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.492 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.407 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.535 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.424 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:10.541 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.427 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:10.523 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.515 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.513+0000 7f25ff5f5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:01.515 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.513+0000 7f25ff5f5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:01.515 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.513+0000 7f25ff5f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:01.515 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.513+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25f8069c00 con 0x7f25f813eca0 2023-12-07T07:33:01.527 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:10.639 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.560 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.558+0000 7f25edffb640 1 -- 172.21.15.97:0/1176010556 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f25e8079ad0 con 0x7f25f813eca0 2023-12-07T07:33:01.561 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.558+0000 7f25edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:01.561 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.558+0000 7f25edffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:01.561 INFO:teuthology.misc.health.smithi097.stdout: 2023-12-07T07:33:01.561 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:32:50.728917+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:01.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 10 librados: watch_flush enter 2023-12-07T07:33:01.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 10 librados: watch_flush exit 2023-12-07T07:33:01.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 20 client.4210.objecter shutdown clearing up homeless session... 2023-12-07T07:33:01.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 10 client.4210.objecter successfully canceled tick 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f25dc06c740 msgr2=0x7f25dc06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f25dc06c740 0x7f25dc06ec00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f25f4005020 tx=0x7f25f4007ef0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 10 monclient: shutdown 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25f813eca0 msgr2=0x7f25f8138f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25f813eca0 0x7f25f8138f60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f25f80aad90 tx=0x7f25e80046c0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 shutdown_connections 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f25dc06c740 0x7f25dc06ec00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25f8146150 0x7f25f81dad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25f813eca0 0x7f25f8138f60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 --2- 172.21.15.97:0/1176010556 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25f80a8a00 0x7f25f8138a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 >> 172.21.15.97:0/1176010556 conn(0x7f25f8066e80 msgr2=0x7f25f813caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:01.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 shutdown_connections 2023-12-07T07:33:01.565 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.560+0000 7f25ff5f5640 1 -- 172.21.15.97:0/1176010556 wait complete. 2023-12-07T07:33:01.565 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:01.561+0000 7f25ff5f5640 1 librados: shutdown 2023-12-07T07:33:01.572 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T07:33:01.610 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:10.706 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.630 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:10.746 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.652 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:10.757 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.655 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:10.762 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.696 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.823 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.720 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:10.836 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.720 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:10.815 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.737 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.865 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.814 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:10.926 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.903 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:10.999 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.929 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:11.045 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.970 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:11.077 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:01.974 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:11.080 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.013 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:11.129 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.013 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:11.108 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.026 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:11.153 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.101 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:11.213 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.192 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:11.288 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.217 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:11.333 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.267 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:11.374 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.270 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:11.376 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.307 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:11.423 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.307 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:11.402 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.321 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:11.449 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.397 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:11.509 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.480 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:11.576 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.514 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:11.630 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.559 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:11.666 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.566 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:11.672 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.606 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:11.701 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.615 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:11.743 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.684 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:11.796 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.821 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:11.938 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.853 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:11.960 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.858 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:11.964 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.876 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:11.992 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:02.984 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:12.096 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.002 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:12.098 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.037 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:12.153 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.091 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:12.187 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.102 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:12.219 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.150 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:12.277 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.152 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:12.258 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.178 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:12.273 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.233 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:12.328 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.248 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:12.376 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.389 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:12.496 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.444 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:12.551 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.503 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:12.614 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.557 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:12.669 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.631 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:12.737 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:03.685 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:12.791 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:07.988 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:17.104 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:08.200 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:17.316 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:08.202 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:17.298 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:08.234 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:17.329 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:08.378 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:08.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.498+0000 7f0651773640 10 monclient: get_monmap_and_config 2023-12-07T07:33:08.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.498+0000 7f0651773640 10 monclient: build_initial_monmap 2023-12-07T07:33:08.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.498+0000 7f0651773640 10 monclient: monmap: 2023-12-07T07:33:08.503 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:08.503 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:08.503 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:33:08.499184+0000 2023-12-07T07:33:08.503 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:33:08.499184+0000 2023-12-07T07:33:08.503 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:08.504 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:08.504 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:08.504 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:08.504 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:08.504 INFO:teuthology.misc.health.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:08.504 INFO:teuthology.misc.health.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:08.504 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:08.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.498+0000 7f0651773640 1 Processor -- start 2023-12-07T07:33:08.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.498+0000 7f0651773640 1 -- start start 2023-12-07T07:33:08.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.498+0000 7f0651773640 10 monclient: init 2023-12-07T07:33:08.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:08.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-12-07T07:33:08.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f064c0a8a00 0x7f064c0a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:08.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient(hunting): picked mon.noname-c con 0x7f064c0a8a00 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:08.506 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f064c145a40 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:08.506 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f064c145e80 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:08.506 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f064c1455b0 con 0x7f064c145e80 2023-12-07T07:33:08.506 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f064c142d40 con 0x7f064c145a40 2023-12-07T07:33:08.506 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:08.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c0aa970 con 0x7f064c0a8a00 2023-12-07T07:33:08.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:08.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.499+0000 7f0651773640 10 monclient(hunting): authenticate will time out at 1310.877846s 2023-12-07T07:33:08.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.500+0000 7f064b7fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f064c145e80 0x7f064c142630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46246/0 (socket says 172.21.15.97:46246) 2023-12-07T07:33:08.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.500+0000 7f064b7fe640 1 -- 172.21.15.97:0/107278903 learned_addr learned my addr 172.21.15.97:0/107278903 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:08.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.500+0000 7f064affd640 1 --2- 172.21.15.97:0/107278903 >> v2:172.21.15.154:3300/0 conn(0x7f064c0a8a00 0x7f064c0a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:08.512 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.501+0000 7f064affd640 10 monclient(hunting): get_auth_request con 0x7f064c0a8a00 auth_method 0 2023-12-07T07:33:08.512 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.501+0000 7f064affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:08.512 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.501+0000 7f064affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:08.512 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.501+0000 7f064affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:08.512 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.502+0000 7f064affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:08.512 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.502+0000 7f064affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.502+0000 7f064affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.503+0000 7f0649ffb640 1 -- 172.21.15.97:0/107278903 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607877210 0 0) 0x7f064c1455b0 con 0x7f064c145e80 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.503+0000 7f0649ffb640 10 monclient(hunting): _negotiate 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.503+0000 7f0649ffb640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.503+0000 7f0649ffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.503+0000 7f0649ffb640 10 monclient(hunting): my global_id is 4182 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.503+0000 7f0649ffb640 1 -- 172.21.15.97:0/107278903 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0634003520 con 0x7f064c145e80 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 10 monclient(hunting): handle_auth_done global_id 4213 payload 274 2023-12-07T07:33:08.513 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 1 -- 172.21.15.97:0/107278903 >> v1:172.21.15.154:6789/0 conn(0x7f064c145a40 legacy=0x7f064c141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:08.514 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 1 -- 172.21.15.97:0/107278903 >> v1:172.21.15.97:6789/0 conn(0x7f064c145e80 legacy=0x7f064c142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:08.514 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 10 monclient: _finish_hunting 0 2023-12-07T07:33:08.514 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 1 monclient: found mon.noname-c 2023-12-07T07:33:08.514 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:08.514 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:08.514 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.505+0000 7f064affd640 1 -- 172.21.15.97:0/107278903 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c0aac50 con 0x7f064c0a8a00 2023-12-07T07:33:08.515 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.512+0000 7f064affd640 1 --2- 172.21.15.97:0/107278903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c0a8a00 0x7f064c0a8de0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f063c0097a0 tx=0x7f063c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcfa643dc1b0898d server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:08.521 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 1 -- 172.21.15.97:0/107278903 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c030610 con 0x7f064c0a8a00 2023-12-07T07:33:08.521 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:08.521 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:08.522 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: dump: 2023-12-07T07:33:08.522 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:08.522 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:08.522 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.522 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.522 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:08.522 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 1 -- 172.21.15.97:0/107278903 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063c030770 con 0x7f064c0a8a00 2023-12-07T07:33:08.523 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 1 -- 172.21.15.97:0/107278903 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c030b00 con 0x7f064c0a8a00 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0649ffb640 10 monclient: dump: 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:08.524 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0651773640 5 monclient: authenticate success, global_id 4213 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0651773640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.517+0000 7f0651773640 4 set_mon_vals no callback set 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 10 monclient: shutdown 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 1 -- 172.21.15.97:0/107278903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c0a8a00 msgr2=0x7f064c0a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 1 --2- 172.21.15.97:0/107278903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c0a8a00 0x7f064c0a8de0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f063c0097a0 tx=0x7f063c02b960 comp rx=0 tx=0).stop 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 1 -- 172.21.15.97:0/107278903 shutdown_connections 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 1 --2- 172.21.15.97:0/107278903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c0a8a00 0x7f064c0a8de0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 1 -- 172.21.15.97:0/107278903 >> 172.21.15.97:0/107278903 conn(0x7f064c066e80 msgr2=0x7f064c0b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:08.525 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 1 -- 172.21.15.97:0/107278903 shutdown_connections 2023-12-07T07:33:08.526 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 1 -- 172.21.15.97:0/107278903 wait complete. 2023-12-07T07:33:08.526 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 10 monclient: build_initial_monmap 2023-12-07T07:33:08.526 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.518+0000 7f0651773640 10 monclient: monmap: 2023-12-07T07:33:08.526 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:08.526 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:08.526 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 librados: starting msgr at 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 librados: starting objecter 2023-12-07T07:33:08.527 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 Processor -- start 2023-12-07T07:33:08.528 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 -- start start 2023-12-07T07:33:08.528 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 librados: setting wanted keys 2023-12-07T07:33:08.528 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 librados: calling monclient init 2023-12-07T07:33:08.528 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 10 monclient: init 2023-12-07T07:33:08.534 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:08.534 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f064c0a8a00 0x7f064c0ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 10 monclient(hunting): picked mon.noname-a con 0x7f064c0a8a00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f064c145a40 0x7f064c070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 10 monclient(hunting): picked mon.noname-c con 0x7f064c145a40 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.519+0000 7f0651773640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c145e80 0x7f064c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 10 monclient(hunting): picked mon.noname-b con 0x7f064c145e80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0ab7d0 con 0x7f064c0a8a00 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c06eb10 con 0x7f064c145e80 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0ac8f0 con 0x7f064c145a40 2023-12-07T07:33:08.535 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f0651773640 10 monclient(hunting): authenticate will time out at 1310.898361s 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c145e80 0x7f064c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064a7fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f064c145a40 0x7f064c070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064affd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f064c0a8a00 0x7f064c0ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064a7fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f064c145a40 0x7f064c070ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57656/0 (socket says 172.21.15.97:57656) 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c145e80 0x7f064c070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:46730/0 (socket says 172.21.15.97:46730) 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064a7fc640 1 -- 172.21.15.97:0/1545855088 learned_addr learned my addr 172.21.15.97:0/1545855088 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064a7fc640 10 monclient(hunting): get_auth_request con 0x7f064c145a40 auth_method 0 2023-12-07T07:33:08.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064b7fe640 10 monclient(hunting): get_auth_request con 0x7f064c145e80 auth_method 0 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.520+0000 7f064b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064affd640 10 monclient(hunting): get_auth_request con 0x7f064c0a8a00 auth_method 0 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:08.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:08.538 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:08.538 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:08.538 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:08.538 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:08.538 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:08.539 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:08.539 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:08.539 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:08.539 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:08.540 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.521+0000 7f064affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:08.540 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2023-12-07T07:33:08.540 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 1 -- 172.21.15.97:0/1545855088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c145e80 msgr2=0x7f064c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:08.540 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c145e80 0x7f064c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:08.541 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 1 -- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f064c0a8a00 msgr2=0x7f064c0ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:08.541 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f064c0a8a00 0x7f064c0ac3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:08.541 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:08.541 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 1 monclient: found mon.noname-c 2023-12-07T07:33:08.541 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:08.541 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:08.541 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.524+0000 7f064a7fc640 1 -- 172.21.15.97:0/1545855088 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c009400 con 0x7f064c145a40 2023-12-07T07:33:08.542 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.527+0000 7f064a7fc640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f064c145a40 0x7f064c070ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f064000c480 tx=0x7f064000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:08.542 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.527+0000 7f06337fe640 10 client.?.objecter ms_handle_connect 0x7f064c145a40 2023-12-07T07:33:08.542 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.527+0000 7f06337fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:08.542 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f06337fe640 1 -- 172.21.15.97:0/1545855088 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06400052d0 con 0x7f064c145a40 2023-12-07T07:33:08.542 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f06337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:08.542 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f06337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f06337fe640 10 monclient: dump: 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:08.543 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f06337fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f06337fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f06337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 5 monclient: authenticate success, global_id 4190 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 10 monclient: _renew_subs 2023-12-07T07:33:08.544 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c06f560 con 0x7f064c145a40 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 10 monclient: _renew_subs 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c0aba70 con 0x7f064c145a40 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.530+0000 7f0651773640 1 librados: init done 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.531+0000 7f0651773640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:08.545 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.531+0000 7f0651773640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:08.546 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.531+0000 7f0651773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:08.546 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.531+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c06ecc0 con 0x7f064c145a40 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.533+0000 7f06337fe640 1 -- 172.21.15.97:0/1545855088 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06400011e0 con 0x7f064c145a40 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.533+0000 7f06337fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.533+0000 7f06337fe640 1 -- 172.21.15.97:0/1545855088 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640012630 con 0x7f064c145a40 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.533+0000 7f06337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.533+0000 7f06337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.533+0000 7f06337fe640 10 monclient: dump: 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:08.548 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:08.549 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:08.549 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:08.549 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:08.549 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.533+0000 7f06497fa640 4 set_mon_vals no callback set 2023-12-07T07:33:08.562 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.558+0000 7f06337fe640 1 -- 172.21.15.97:0/1545855088 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f0640012790 con 0x7f064c145a40 2023-12-07T07:33:08.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.559+0000 7f06337fe640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f063806c740 0x7f063806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:08.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.559+0000 7f064affd640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f063806c740 0x7f063806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:08.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.559+0000 7f064affd640 10 monclient: get_auth_request con 0x7f063806c740 auth_method 0 2023-12-07T07:33:08.563 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.560+0000 7f06337fe640 10 client.4190.objecter ms_handle_connect 0x7f063806c740 2023-12-07T07:33:08.564 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.560+0000 7f064affd640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f063806c740 0x7f063806ec00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f063c036000 tx=0x7f063c03b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 1 -- 172.21.15.97:0/1545855088 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0640058770 con 0x7f064c145a40 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 10 client.4190.objecter ms_dispatch 0x7f064c000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 20 client.4190.objecter dump_active .. 0 homeless 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 1 -- 172.21.15.97:0/1545855088 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0640058540 con 0x7f064c145a40 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:08.969 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:08.966+0000 7f06337fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:08.991 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:18.118 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:09.045 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:18.152 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:09.187 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.185+0000 7f0651773640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:09.187 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.185+0000 7f0651773640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:09.187 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.185+0000 7f0651773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:09.187 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.185+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f064c0aba70 con 0x7f064c145a40 2023-12-07T07:33:09.211 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.209+0000 7f06337fe640 1 -- 172.21.15.97:0/1545855088 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f064c0aba70 con 0x7f064c145a40 2023-12-07T07:33:09.211 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.209+0000 7f06337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:09.211 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.209+0000 7f06337fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:09.211 INFO:teuthology.misc.health.smithi097.stdout: 2023-12-07T07:33:09.212 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:32:50.728917+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 10 librados: watch_flush enter 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 10 librados: watch_flush exit 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 20 client.4190.objecter shutdown clearing up homeless session... 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 10 client.4190.objecter successfully canceled tick 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f063806c740 msgr2=0x7f063806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f063806c740 0x7f063806ec00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f063c036000 tx=0x7f063c03b040 comp rx=0 tx=0).stop 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 10 monclient: shutdown 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f064c145a40 msgr2=0x7f064c070ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.211+0000 7f0651773640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f064c145a40 0x7f064c070ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f064000c480 tx=0x7f064000c940 comp rx=0 tx=0).stop 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 shutdown_connections 2023-12-07T07:33:09.215 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f063806c740 0x7f063806ec00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:09.216 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064c145e80 0x7f064c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:09.216 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f064c145a40 0x7f064c070ab0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:09.216 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 --2- 172.21.15.97:0/1545855088 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f064c0a8a00 0x7f064c0ac3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:09.216 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 >> 172.21.15.97:0/1545855088 conn(0x7f064c066e80 msgr2=0x7f064c141570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:09.216 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 shutdown_connections 2023-12-07T07:33:09.216 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 -- 172.21.15.97:0/1545855088 wait complete. 2023-12-07T07:33:09.216 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:09.212+0000 7f0651773640 1 librados: shutdown 2023-12-07T07:33:09.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T07:33:09.698 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:18.804 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:09.818 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:18.930 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:13.526 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:22.642 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:13.529 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:22.646 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:13.646 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:22.763 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:13.653 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:22.769 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:13.864 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:22.959 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:13.868 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:22.963 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:14.184 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:23.281 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:14.190 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:23.285 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:14.896 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:24.003 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:14.900 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:24.008 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:15.089 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:24.216 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:15.095 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:24.223 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:15.651 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:24.757 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:15.655 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:24.761 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:15.808 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:24.920 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:15.814 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:24.926 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:16.032 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.133+0000 7f397431d640 10 monclient: get_monmap_and_config 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.133+0000 7f397431d640 10 monclient: build_initial_monmap 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.133+0000 7f397431d640 10 monclient: monmap: 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:33:16.134616+0000 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:33:16.134616+0000 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:16.136 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:16.137 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:16.137 INFO:teuthology.misc.health.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:16.137 INFO:teuthology.misc.health.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:16.137 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:16.137 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.134+0000 7f397431d640 1 Processor -- start 2023-12-07T07:33:16.137 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.134+0000 7f397431d640 1 -- start start 2023-12-07T07:33:16.137 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.134+0000 7f397431d640 10 monclient: init 2023-12-07T07:33:16.138 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:16.138 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f396c0a89e0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f396c13f630 0x7f396c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient(hunting): picked mon.noname-a con 0x7f396c13f630 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f396c142710 0x7f396c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient(hunting): picked mon.noname-c con 0x7f396c142710 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c0aa770 con 0x7f396c0a89e0 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c0aa9c0 con 0x7f396c13f630 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c0aaca0 con 0x7f396c142710 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f397431d640 10 monclient(hunting): authenticate will time out at 1318.513560s 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.135+0000 7f39728ba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f396c142710 0x7f396c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f396c142710 0x7f396c14c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:60988/0 (socket says 172.21.15.97:60988) 2023-12-07T07:33:16.139 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 1 -- 172.21.15.97:0/2774852820 learned_addr learned my addr 172.21.15.97:0/2774852820 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:16.140 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 10 monclient(hunting): get_auth_request con 0x7f396c142710 auth_method 0 2023-12-07T07:33:16.140 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:16.140 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:16.140 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:16.140 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:16.140 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:16.140 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.136+0000 7f39728ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:16.141 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 10 monclient(hunting): handle_auth_done global_id 4222 payload 274 2023-12-07T07:33:16.141 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 1 -- 172.21.15.97:0/2774852820 >> v2:172.21.15.97:3300/0 conn(0x7f396c13f630 msgr2=0x7f396c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:16.141 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 1 --2- 172.21.15.97:0/2774852820 >> v2:172.21.15.97:3300/0 conn(0x7f396c13f630 0x7f396c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.141 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 1 -- 172.21.15.97:0/2774852820 >> v1:172.21.15.97:6789/0 conn(0x7f396c0a89e0 legacy=0x7f396c13ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:16.141 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 10 monclient: _finish_hunting 0 2023-12-07T07:33:16.141 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 1 monclient: found mon.noname-c 2023-12-07T07:33:16.141 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:16.142 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:16.142 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.139+0000 7f39728ba640 1 -- 172.21.15.97:0/2774852820 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c0aaf80 con 0x7f396c142710 2023-12-07T07:33:16.147 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.144+0000 7f39728ba640 1 --2- 172.21.15.97:0/2774852820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c142710 0x7f396c14c800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f396c0a84a0 tx=0x7f396402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e75b33b50bd650c5 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:16.223 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 1 -- 172.21.15.97:0/2774852820 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396402c9f0 con 0x7f396c142710 2023-12-07T07:33:16.223 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:16.223 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:16.223 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: dump: 2023-12-07T07:33:16.223 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:16.223 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:16.224 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: _finish_auth 0 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: _check_auth_tickets 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 1 -- 172.21.15.97:0/2774852820 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396402cb50 con 0x7f396c142710 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f397431d640 5 monclient: authenticate success, global_id 4222 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f397431d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 1 -- 172.21.15.97:0/2774852820 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640343f0 con 0x7f396c142710 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f39710b7640 10 monclient: dump: 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:16.225 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f397431d640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.217+0000 7f397431d640 4 set_mon_vals no callback set 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 10 monclient: shutdown 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 -- 172.21.15.97:0/2774852820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c142710 msgr2=0x7f396c14c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 --2- 172.21.15.97:0/2774852820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c142710 0x7f396c14c800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f396c0a84a0 tx=0x7f396402be50 comp rx=0 tx=0).stop 2023-12-07T07:33:16.226 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 -- 172.21.15.97:0/2774852820 shutdown_connections 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 --2- 172.21.15.97:0/2774852820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c142710 0x7f396c14c800 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 --2- 172.21.15.97:0/2774852820 >> v2:172.21.15.97:3300/0 conn(0x7f396c13f630 0x7f396c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 -- 172.21.15.97:0/2774852820 >> 172.21.15.97:0/2774852820 conn(0x7f396c0b7940 msgr2=0x7f396c0b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 -- 172.21.15.97:0/2774852820 shutdown_connections 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.218+0000 7f397431d640 1 -- 172.21.15.97:0/2774852820 wait complete. 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 10 monclient: build_initial_monmap 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 10 monclient: monmap: 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:16.227 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 1 librados: starting msgr at 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 1 librados: starting objecter 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 1 Processor -- start 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 1 -- start start 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 1 librados: setting wanted keys 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 1 librados: calling monclient init 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.219+0000 7f397431d640 10 monclient: init 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c0a89e0 0x7f396c0b6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): picked mon.noname-b con 0x7f396c0a89e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f396c13f630 0x7f396c145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): picked mon.noname-c con 0x7f396c13f630 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:16.228 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c142710 0x7f396c1d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): picked mon.noname-a con 0x7f396c142710 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c145df0 con 0x7f396c142710 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c146210 con 0x7f396c0a89e0 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c0ac1c0 con 0x7f396c13f630 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f397431d640 10 monclient(hunting): authenticate will time out at 1318.598535s 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.220+0000 7f39718b8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f396c13f630 0x7f396c145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.221+0000 7f39718b8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f396c13f630 0x7f396c145690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57932/0 (socket says 172.21.15.97:57932) 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.221+0000 7f39718b8640 1 -- 172.21.15.97:0/2740892772 learned_addr learned my addr 172.21.15.97:0/2740892772 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.221+0000 7f39718b8640 10 monclient(hunting): get_auth_request con 0x7f396c13f630 auth_method 0 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.221+0000 7f39718b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:16.229 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.221+0000 7f39718b8640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:16.230 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.221+0000 7f39718b8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:16.231 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.222+0000 7f39718b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:16.231 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.222+0000 7f39718b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:16.231 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.222+0000 7f39718b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:16.231 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.222+0000 7f39728ba640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c142710 0x7f396c1d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.222+0000 7f39720b9640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c0a89e0 0x7f396c0b6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39720b9640 10 monclient(hunting): get_auth_request con 0x7f396c0a89e0 auth_method 0 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39720b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39720b9640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39720b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39728ba640 10 monclient(hunting): get_auth_request con 0x7f396c142710 auth_method 0 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39728ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39728ba640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:16.232 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39728ba640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39720b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39720b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39720b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39728ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39728ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.223+0000 7f39728ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 1 -- 172.21.15.97:0/2740892772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c0a89e0 msgr2=0x7f396c0b6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c0a89e0 0x7f396c0b6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.233 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 1 -- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c142710 msgr2=0x7f396c1d6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c142710 0x7f396c1d6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 10 monclient: _finish_hunting 0 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 1 monclient: found mon.noname-c 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.225+0000 7f39718b8640 1 -- 172.21.15.97:0/2740892772 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964009b30 con 0x7f396c13f630 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.227+0000 7f39718b8640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f396c13f630 0x7f396c145690 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f395c00b4d0 tx=0x7f395c00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:16.234 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.227+0000 7f3962ffd640 10 client.?.objecter ms_handle_connect 0x7f396c13f630 2023-12-07T07:33:16.236 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.227+0000 7f3962ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:16.241 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 1 -- 172.21.15.97:0/2740892772 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c010870 con 0x7f396c13f630 2023-12-07T07:33:16.241 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:16.241 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:16.241 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: dump: 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:16.242 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:16.243 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:16.243 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:16.245 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:16.245 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:16.245 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:16.245 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 1 -- 172.21.15.97:0/2740892772 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c0109d0 con 0x7f396c13f630 2023-12-07T07:33:16.245 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 5 monclient: authenticate success, global_id 4193 2023-12-07T07:33:16.246 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:16.246 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 10 monclient: _renew_subs 2023-12-07T07:33:16.246 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 1 -- 172.21.15.97:0/2740892772 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c010d80 con 0x7f396c13f630 2023-12-07T07:33:16.246 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:16.246 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c145190 con 0x7f396c13f630 2023-12-07T07:33:16.246 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:16.247 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:16.247 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f3962ffd640 10 monclient: dump: 2023-12-07T07:33:16.247 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:16.247 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:16.250 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.250 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:16.250 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:16.250 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f39708b6640 4 set_mon_vals no callback set 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 10 monclient: _renew_subs 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c1d7680 con 0x7f396c13f630 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.232+0000 7f397431d640 1 librados: init done 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.233+0000 7f397431d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:16.251 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.233+0000 7f397431d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:16.252 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.233+0000 7f397431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:16.252 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.233+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c13da10 con 0x7f396c13f630 2023-12-07T07:33:16.253 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.251+0000 7f3962ffd640 1 -- 172.21.15.97:0/2740892772 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f395c023480 con 0x7f396c13f630 2023-12-07T07:33:16.254 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.252+0000 7f3962ffd640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f395006c740 0x7f395006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:16.255 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.252+0000 7f39720b9640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f395006c740 0x7f395006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:16.255 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.252+0000 7f39720b9640 10 monclient: get_auth_request con 0x7f395006c740 auth_method 0 2023-12-07T07:33:16.255 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.252+0000 7f3962ffd640 10 client.4193.objecter ms_handle_connect 0x7f395006c740 2023-12-07T07:33:16.255 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.252+0000 7f39720b9640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f395006c740 0x7f395006ec00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3968001840 tx=0x7f3968009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:16.326 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.321+0000 7f3962ffd640 1 -- 172.21.15.97:0/2740892772 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f395c058740 con 0x7f396c13f630 2023-12-07T07:33:16.326 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.321+0000 7f3962ffd640 10 client.4193.objecter ms_dispatch 0x7f396c000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T07:33:16.326 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.321+0000 7f3962ffd640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T07:33:16.326 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.321+0000 7f3962ffd640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2023-12-07T07:33:16.326 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.321+0000 7f3962ffd640 20 client.4193.objecter dump_active .. 0 homeless 2023-12-07T07:33:16.600 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.598+0000 7f3962ffd640 1 -- 172.21.15.97:0/2740892772 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f395c054a80 con 0x7f396c13f630 2023-12-07T07:33:16.601 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.598+0000 7f3962ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:16.601 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.598+0000 7f3962ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:16.861 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.859+0000 7f397431d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:16.862 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.859+0000 7f397431d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:16.862 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.859+0000 7f397431d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:16.862 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.859+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f396c069c00 con 0x7f396c13f630 2023-12-07T07:33:16.879 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.877+0000 7f3962ffd640 1 -- 172.21.15.97:0/2740892772 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f395c01e020 con 0x7f396c13f630 2023-12-07T07:33:16.879 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.877+0000 7f3962ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:16.879 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.877+0000 7f3962ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:16.880 INFO:teuthology.misc.health.smithi097.stdout: 2023-12-07T07:33:16.880 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:32:50.728917+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:16.881 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.879+0000 7f397431d640 10 librados: watch_flush enter 2023-12-07T07:33:16.882 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 10 librados: watch_flush exit 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 20 client.4193.objecter shutdown clearing up homeless session... 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 10 client.4193.objecter successfully canceled tick 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f395006c740 msgr2=0x7f395006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f395006c740 0x7f395006ec00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3968001840 tx=0x7f3968009040 comp rx=0 tx=0).stop 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 10 monclient: shutdown 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f396c13f630 msgr2=0x7f396c145690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f396c13f630 0x7f396c145690 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f395c00b4d0 tx=0x7f395c00b990 comp rx=0 tx=0).stop 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 shutdown_connections 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f395006c740 0x7f395006ec00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c142710 0x7f396c1d6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f396c13f630 0x7f396c145690 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f395c00b4d0 tx=0x7f395c00b990 comp rx=0 tx=0).stop 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 --2- 172.21.15.97:0/2740892772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c0a89e0 0x7f396c0b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:16.883 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 >> 172.21.15.97:0/2740892772 conn(0x7f396c0b7940 msgr2=0x7f396c0b2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:16.884 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.880+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 shutdown_connections 2023-12-07T07:33:16.884 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.881+0000 7f397431d640 1 -- 172.21.15.97:0/2740892772 wait complete. 2023-12-07T07:33:16.884 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:16.881+0000 7f397431d640 1 librados: shutdown 2023-12-07T07:33:16.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T07:33:17.209 INFO:tasks.ceph.osd.5.smithi154.stderr:==00:00:00:26.325 86723== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:17.501 INFO:tasks.ceph.osd.4.smithi154.stderr:==00:00:00:26.618 86724== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:17.636 INFO:tasks.ceph.osd.7.smithi154.stderr:==00:00:00:26.731 86726== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:18.029 INFO:tasks.ceph.osd.6.smithi154.stderr:==00:00:00:27.125 86725== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:19.678 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:28.803 87060== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:19.706 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:28.813 87063== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:19.927 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:29.039 87061== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:20.231 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:29.337 87062== Warning: unimplemented fcntl command: 1038 2023-12-07T07:33:23.701 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:23.788 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.786+0000 7f8df6141640 10 monclient: get_monmap_and_config 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.786+0000 7f8df6141640 10 monclient: build_initial_monmap 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.786+0000 7f8df6141640 10 monclient: monmap: 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:33:23.787730+0000 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:33:23.787730+0000 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:23.789 INFO:teuthology.misc.health.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:23.790 INFO:teuthology.misc.health.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:23.790 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:23.792 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.787+0000 7f8df6141640 1 Processor -- start 2023-12-07T07:33:23.792 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.787+0000 7f8df6141640 1 -- start start 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.787+0000 7f8df6141640 10 monclient: init 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8df00a89e0 0x7f8df013ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient(hunting): picked mon.noname-a con 0x7f8df00a89e0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8df013f460 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8df01428e0 0x7f8df014c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient(hunting): picked mon.noname-c con 0x7f8df01428e0 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00aa770 con 0x7f8df013f460 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df013f860 con 0x7f8df00a89e0 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df00aa9c0 con 0x7f8df01428e0 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:23.793 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8df6141640 10 monclient(hunting): authenticate will time out at 1326.166616s 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.788+0000 7f8def7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8df00a89e0 0x7f8df013ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8deffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8df01428e0 0x7f8df014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8def7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8df00a89e0 0x7f8df013ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:51666/0 (socket says 172.21.15.97:51666) 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8def7fe640 1 -- 172.21.15.97:0/1304811139 learned_addr learned my addr 172.21.15.97:0/1304811139 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8def7fe640 10 monclient(hunting): get_auth_request con 0x7f8df00a89e0 auth_method 0 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8def7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8def7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8def7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8deffff640 10 monclient(hunting): get_auth_request con 0x7f8df01428e0 auth_method 0 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8deffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.789+0000 7f8deffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.790+0000 7f8deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.790+0000 7f8deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:23.794 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.790+0000 7f8deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:23.795 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.790+0000 7f8def7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:23.795 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.790+0000 7f8def7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:23.795 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.790+0000 7f8def7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 10 monclient(hunting): handle_auth_done global_id 4228 payload 274 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 1 -- 172.21.15.97:0/1304811139 >> v2:172.21.15.97:3300/0 conn(0x7f8df00a89e0 msgr2=0x7f8df013ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 1 --2- 172.21.15.97:0/1304811139 >> v2:172.21.15.97:3300/0 conn(0x7f8df00a89e0 0x7f8df013ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 1 -- 172.21.15.97:0/1304811139 >> v1:172.21.15.97:6789/0 conn(0x7f8df013f460 legacy=0x7f8df01421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 1 monclient: found mon.noname-c 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:23.796 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.792+0000 7f8deffff640 1 -- 172.21.15.97:0/1304811139 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df00aaca0 con 0x7f8df01428e0 2023-12-07T07:33:23.797 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.795+0000 7f8deffff640 1 --2- 172.21.15.97:0/1304811139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df01428e0 0x7f8df014c800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8df00a84a0 tx=0x7f8de802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d18043ad31e6a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:23.824 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 1 -- 172.21.15.97:0/1304811139 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de802cbf0 con 0x7f8df01428e0 2023-12-07T07:33:23.824 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:23.824 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:23.824 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: dump: 2023-12-07T07:33:23.824 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:23.825 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:23.825 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.825 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.825 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:23.825 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:23.825 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:23.825 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:23.826 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:23.826 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:23.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:23.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:23.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:23.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 1 -- 172.21.15.97:0/1304811139 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de802cd50 con 0x7f8df01428e0 2023-12-07T07:33:23.826 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 1 -- 172.21.15.97:0/1304811139 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8034660 con 0x7f8df01428e0 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8dee7fc640 10 monclient: dump: 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.827 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8df6141640 5 monclient: authenticate success, global_id 4228 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8df6141640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.801+0000 7f8df6141640 4 set_mon_vals no callback set 2023-12-07T07:33:23.828 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 10 monclient: shutdown 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 -- 172.21.15.97:0/1304811139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df01428e0 msgr2=0x7f8df014c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 --2- 172.21.15.97:0/1304811139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df01428e0 0x7f8df014c800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8df00a84a0 tx=0x7f8de802be40 comp rx=0 tx=0).stop 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 -- 172.21.15.97:0/1304811139 shutdown_connections 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 --2- 172.21.15.97:0/1304811139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df01428e0 0x7f8df014c800 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 --2- 172.21.15.97:0/1304811139 >> v2:172.21.15.97:3300/0 conn(0x7f8df00a89e0 0x7f8df013ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 -- 172.21.15.97:0/1304811139 >> 172.21.15.97:0/1304811139 conn(0x7f8df00b7940 msgr2=0x7f8df00b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 -- 172.21.15.97:0/1304811139 shutdown_connections 2023-12-07T07:33:23.829 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.802+0000 7f8df6141640 1 -- 172.21.15.97:0/1304811139 wait complete. 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.803+0000 7f8df6141640 10 monclient: build_initial_monmap 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.803+0000 7f8df6141640 10 monclient: monmap: 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:23.830 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.803+0000 7f8df6141640 1 librados: starting msgr at 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.803+0000 7f8df6141640 1 librados: starting objecter 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.804+0000 7f8df6141640 1 Processor -- start 2023-12-07T07:33:23.831 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.804+0000 7f8df6141640 1 -- start start 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.804+0000 7f8df6141640 1 librados: setting wanted keys 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.804+0000 7f8df6141640 1 librados: calling monclient init 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.804+0000 7f8df6141640 10 monclient: init 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8df00a89e0 0x7f8df0144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 10 monclient(hunting): picked mon.noname-c con 0x7f8df00a89e0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:23.832 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df013f460 0x7f8df00acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:23.833 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 10 monclient(hunting): picked mon.noname-b con 0x7f8df013f460 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:23.833 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8df01428e0 0x7f8df00ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:23.833 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8df6141640 10 monclient(hunting): picked mon.noname-a con 0x7f8df01428e0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:23.833 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.805+0000 7f8def7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8df00a89e0 0x7f8df0144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:23.833 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8df6141640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:23.833 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8df6141640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df00ad720 con 0x7f8df01428e0 2023-12-07T07:33:23.833 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8deffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8df01428e0 0x7f8df00ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:23.834 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8def7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8df00a89e0 0x7f8df0144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:50578/0 (socket says 172.21.15.97:50578) 2023-12-07T07:33:23.834 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8def7fe640 1 -- 172.21.15.97:0/1618522896 learned_addr learned my addr 172.21.15.97:0/1618522896 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:23.834 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8deeffd640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df013f460 0x7f8df00acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:23.834 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8df6141640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:23.834 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df00aee70 con 0x7f8df013f460 2023-12-07T07:33:23.834 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8df6141640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:23.835 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.806+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df00ad3e0 con 0x7f8df00a89e0 2023-12-07T07:33:23.835 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8df6141640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:23.835 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8df6141640 10 monclient(hunting): authenticate will time out at 1326.185577s 2023-12-07T07:33:23.835 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8def7fe640 10 monclient(hunting): get_auth_request con 0x7f8df00a89e0 auth_method 0 2023-12-07T07:33:23.835 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8def7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:23.835 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8def7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:23.835 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8def7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deeffd640 10 monclient(hunting): get_auth_request con 0x7f8df013f460 auth_method 0 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deeffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deeffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deffff640 10 monclient(hunting): get_auth_request con 0x7f8df01428e0 auth_method 0 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:23.836 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:23.837 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:23.837 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.807+0000 7f8deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:23.837 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.808+0000 7f8deeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:23.837 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.808+0000 7f8deeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:23.837 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.808+0000 7f8deeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:23.837 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.808+0000 7f8def7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.808+0000 7f8def7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.808+0000 7f8def7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.810+0000 7f8def7fe640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.810+0000 7f8def7fe640 1 -- 172.21.15.97:0/1618522896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df013f460 msgr2=0x7f8df00acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.810+0000 7f8def7fe640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df013f460 0x7f8df00acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.811+0000 7f8def7fe640 1 -- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8df01428e0 msgr2=0x7f8df00ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.811+0000 7f8def7fe640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8df01428e0 0x7f8df00ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.811+0000 7f8def7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.811+0000 7f8def7fe640 1 monclient: found mon.noname-c 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.811+0000 7f8def7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.811+0000 7f8def7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.811+0000 7f8def7fe640 1 -- 172.21.15.97:0/1618522896 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de8009a70 con 0x7f8df00a89e0 2023-12-07T07:33:23.838 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.813+0000 7f8dd7fff640 10 client.?.objecter ms_handle_connect 0x7f8df00a89e0 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.813+0000 7f8dd7fff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.813+0000 7f8def7fe640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8df00a89e0 0x7f8df0144ee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8de000e870 tx=0x7f8de000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 1 -- 172.21.15.97:0/1618522896 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000c8e0 con 0x7f8df00a89e0 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: dump: 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:23.839 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: _finish_auth 0 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 1 -- 172.21.15.97:0/1618522896 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de000ca40 con 0x7f8df00a89e0 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 1 -- 172.21.15.97:0/1618522896 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000cdf0 con 0x7f8df00a89e0 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dd7fff640 10 monclient: dump: 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:23.840 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.816+0000 7f8dedffb640 4 set_mon_vals no callback set 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 5 monclient: authenticate success, global_id 4196 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 10 monclient: _renew_subs 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df00adbb0 con 0x7f8df00a89e0 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 10 monclient: _renew_subs 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df00ab970 con 0x7f8df00a89e0 2023-12-07T07:33:23.841 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.817+0000 7f8df6141640 1 librados: init done 2023-12-07T07:33:23.842 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.819+0000 7f8df6141640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:23.842 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.819+0000 7f8df6141640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:23.842 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.819+0000 7f8df6141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:23.842 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:23.819+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df0069c00 con 0x7f8df00a89e0 2023-12-07T07:33:24.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.045+0000 7f8dd7fff640 1 -- 172.21.15.97:0/1618522896 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f8de0013470 con 0x7f8df00a89e0 2023-12-07T07:33:24.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.046+0000 7f8dd7fff640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8ddc06c790 0x7f8ddc06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:24.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.046+0000 7f8deeffd640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8ddc06c790 0x7f8ddc06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:24.048 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.046+0000 7f8deeffd640 10 monclient: get_auth_request con 0x7f8ddc06c790 auth_method 0 2023-12-07T07:33:24.049 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.046+0000 7f8deeffd640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8ddc06c790 0x7f8ddc06ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8df01457a0 tx=0x7f8de4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:24.049 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.046+0000 7f8dd7fff640 10 client.4196.objecter ms_handle_connect 0x7f8ddc06c790 2023-12-07T07:33:24.243 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.241+0000 7f8dd7fff640 1 -- 172.21.15.97:0/1618522896 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8de0058d00 con 0x7f8df00a89e0 2023-12-07T07:33:24.243 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.241+0000 7f8dd7fff640 10 client.4196.objecter ms_dispatch 0x7f8df0000b90 osd_map(10..10 src has 1..10) v4 2023-12-07T07:33:24.243 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.241+0000 7f8dd7fff640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2023-12-07T07:33:24.243 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.241+0000 7f8dd7fff640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2023-12-07T07:33:24.243 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.241+0000 7f8dd7fff640 20 client.4196.objecter dump_active .. 0 homeless 2023-12-07T07:33:24.255 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.253+0000 7f8dd7fff640 1 -- 172.21.15.97:0/1618522896 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8de00557d0 con 0x7f8df00a89e0 2023-12-07T07:33:24.255 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.253+0000 7f8dd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:24.255 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.253+0000 7f8dd7fff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:24.536 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.534+0000 7f8df6141640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:24.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.534+0000 7f8df6141640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:24.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.534+0000 7f8df6141640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:24.537 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.534+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8df00ab010 con 0x7f8df00a89e0 2023-12-07T07:33:24.565 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.562+0000 7f8dd7fff640 1 -- 172.21.15.97:0/1618522896 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8de0017020 con 0x7f8df00a89e0 2023-12-07T07:33:24.565 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.562+0000 7f8dd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:24.565 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.562+0000 7f8dd7fff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:24.566 INFO:teuthology.misc.health.smithi097.stdout: 2023-12-07T07:33:24.566 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:32:50.728917+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:24.568 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 10 librados: watch_flush enter 2023-12-07T07:33:24.568 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 10 librados: watch_flush exit 2023-12-07T07:33:24.568 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 20 client.4196.objecter shutdown clearing up homeless session... 2023-12-07T07:33:24.568 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 10 client.4196.objecter successfully canceled tick 2023-12-07T07:33:24.568 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8ddc06c790 msgr2=0x7f8ddc06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:24.568 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8ddc06c790 0x7f8ddc06ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8df01457a0 tx=0x7f8de4009040 comp rx=0 tx=0).stop 2023-12-07T07:33:24.568 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 10 monclient: shutdown 2023-12-07T07:33:24.569 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:24.569 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8df00a89e0 msgr2=0x7f8df0144ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:24.569 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.565+0000 7f8df6141640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8df00a89e0 0x7f8df0144ee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8de000e870 tx=0x7f8de000ed30 comp rx=0 tx=0).stop 2023-12-07T07:33:24.570 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.566+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 shutdown_connections 2023-12-07T07:33:24.570 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.566+0000 7f8df6141640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8ddc06c790 0x7f8ddc06ec50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:24.570 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.566+0000 7f8df6141640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8df01428e0 0x7f8df00ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:24.570 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.566+0000 7f8df6141640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8df013f460 0x7f8df00acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:24.571 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.566+0000 7f8df6141640 1 --2- 172.21.15.97:0/1618522896 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8df00a89e0 0x7f8df0144ee0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:24.571 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.566+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 >> 172.21.15.97:0/1618522896 conn(0x7f8df00b7940 msgr2=0x7f8df013bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:24.571 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.567+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 shutdown_connections 2023-12-07T07:33:24.571 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.567+0000 7f8df6141640 1 -- 172.21.15.97:0/1618522896 wait complete. 2023-12-07T07:33:24.571 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:24.567+0000 7f8df6141640 1 librados: shutdown 2023-12-07T07:33:24.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-07T07:33:25.179 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-12-07T07:33:25.174+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-12-07T07:33:25.787 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-12-07T07:33:25.781+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-12-07T07:33:25.987 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:33:25.982+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-12-07T07:33:26.330 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-12-07T07:33:26.259+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-12-07T07:33:28.266 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-12-07T07:33:28.260+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-12-07T07:33:28.283 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:33:28.277+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-12-07T07:33:28.438 INFO:tasks.ceph.osd.3.smithi097.stderr:2023-12-07T07:33:28.433+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-12-07T07:33:28.926 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-12-07T07:33:28.920+0000 ac7d640 -1 osd.4 0 waiting for initial osdmap 2023-12-07T07:33:28.935 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:33:28.922+0000 ac7d640 -1 osd.7 0 waiting for initial osdmap 2023-12-07T07:33:28.951 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-12-07T07:33:28.946+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-12-07T07:33:28.962 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-12-07T07:33:28.956+0000 c493640 -1 osd.5 0 waiting for initial osdmap 2023-12-07T07:33:29.322 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-12-07T07:33:29.301+0000 1425b640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:29.359 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-12-07T07:33:29.341+0000 1425b640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:29.398 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:33:29.377+0000 1425b640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:30.092 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-12-07T07:33:30.086+0000 c493640 -1 osd.6 0 waiting for initial osdmap 2023-12-07T07:33:30.490 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-12-07T07:33:30.475+0000 1425b640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:31.391 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:31.392 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-12-07T07:33:31.384+0000 c493640 -1 osd.0 0 waiting for initial osdmap 2023-12-07T07:33:31.481 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.479+0000 7f21dfc75640 10 monclient: get_monmap_and_config 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.479+0000 7f21dfc75640 10 monclient: build_initial_monmap 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.479+0000 7f21dfc75640 10 monclient: monmap: 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:33:31.480662+0000 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:33:31.480662+0000 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:31.482 INFO:teuthology.misc.health.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:31.483 INFO:teuthology.misc.health.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:31.483 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 1 Processor -- start 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 1 -- start start 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient: init 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21d80a8a00 0x7f21d80a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient(hunting): picked mon.noname-c con 0x7f21d80a8a00 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21d8145a40 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f21d8145e80 0x7f21d8142630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient(hunting): picked mon.noname-a con 0x7f21d8145e80 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d81455b0 con 0x7f21d8145a40 2023-12-07T07:33:31.485 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80aa970 con 0x7f21d8145e80 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80aac50 con 0x7f21d80a8a00 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.480+0000 7f21dfc75640 10 monclient(hunting): authenticate will time out at 1333.859253s 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.481+0000 7f21dda11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21d80a8a00 0x7f21d80a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.481+0000 7f21de212640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f21d8145e80 0x7f21d8142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.481+0000 7f21dd210640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f21d8145a40 0x7f21d8141f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39554/0 (socket says 172.21.15.97:39554) 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.481+0000 7f21dd210640 1 -- 172.21.15.97:0/376014009 learned_addr learned my addr 172.21.15.97:0/376014009 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21dda11640 10 monclient(hunting): get_auth_request con 0x7f21d80a8a00 auth_method 0 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21dda11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21dda11640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:31.486 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21dda11640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:31.487 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21de212640 10 monclient(hunting): get_auth_request con 0x7f21d8145e80 auth_method 0 2023-12-07T07:33:31.487 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21de212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:31.487 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21de212640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:31.487 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21de212640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:31.487 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21dda11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:31.487 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21dda11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:31.487 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.482+0000 7f21dda11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21de212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21de212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21de212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 1 -- 172.21.15.97:0/376014009 >> v2:172.21.15.97:3300/0 conn(0x7f21d8145e80 msgr2=0x7f21d8142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 1 --2- 172.21.15.97:0/376014009 >> v2:172.21.15.97:3300/0 conn(0x7f21d8145e80 0x7f21d8142630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 1 -- 172.21.15.97:0/376014009 >> v1:172.21.15.97:6789/0 conn(0x7f21d8145a40 legacy=0x7f21d8141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 10 monclient: _finish_hunting 0 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 1 monclient: found mon.noname-c 2023-12-07T07:33:31.488 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:31.489 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:31.489 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.484+0000 7f21dda11640 1 -- 172.21.15.97:0/376014009 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80aaf30 con 0x7f21d80a8a00 2023-12-07T07:33:31.489 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.487+0000 7f21dda11640 1 --2- 172.21.15.97:0/376014009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 0x7f21d80a8de0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21d80b53d0 tx=0x7f21d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4218f0775e43b1cb server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 1 -- 172.21.15.97:0/376014009 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4031830 con 0x7f21d80a8a00 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: dump: 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.493 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: _finish_auth 0 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: _check_auth_tickets 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 1 -- 172.21.15.97:0/376014009 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d4031990 con 0x7f21d80a8a00 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 1 -- 172.21.15.97:0/376014009 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4031d20 con 0x7f21d80a8a00 2023-12-07T07:33:31.494 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dca0f640 10 monclient: dump: 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:31.495 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dfc75640 5 monclient: authenticate success, global_id 4258 2023-12-07T07:33:31.496 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dfc75640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:31.496 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.490+0000 7f21dfc75640 4 set_mon_vals no callback set 2023-12-07T07:33:31.499 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 10 monclient: shutdown 2023-12-07T07:33:31.499 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:31.499 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 -- 172.21.15.97:0/376014009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 msgr2=0x7f21d80a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 --2- 172.21.15.97:0/376014009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 0x7f21d80a8de0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21d80b53d0 tx=0x7f21d402b8d0 comp rx=0 tx=0).stop 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 -- 172.21.15.97:0/376014009 shutdown_connections 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 --2- 172.21.15.97:0/376014009 >> v2:172.21.15.97:3300/0 conn(0x7f21d8145e80 0x7f21d8142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 --2- 172.21.15.97:0/376014009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 0x7f21d80a8de0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 -- 172.21.15.97:0/376014009 >> 172.21.15.97:0/376014009 conn(0x7f21d8066e80 msgr2=0x7f21d80b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 -- 172.21.15.97:0/376014009 shutdown_connections 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.491+0000 7f21dfc75640 1 -- 172.21.15.97:0/376014009 wait complete. 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 10 monclient: build_initial_monmap 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 10 monclient: monmap: 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:31.500 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 1 librados: starting msgr at 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 1 librados: starting objecter 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 1 Processor -- start 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 1 -- start start 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 1 librados: setting wanted keys 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 1 librados: calling monclient init 2023-12-07T07:33:31.501 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.492+0000 7f21dfc75640 10 monclient: init 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 0x7f21d80adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): picked mon.noname-b con 0x7f21d80a8a00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21d8145a40 0x7f21d80aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): picked mon.noname-c con 0x7f21d8145a40 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21d8145e80 0x7f21d81d7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): picked mon.noname-a con 0x7f21d8145e80 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d81d6fc0 con 0x7f21d8145e80 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:31.502 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80aeff0 con 0x7f21d80a8a00 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80affb0 con 0x7f21d8145a40 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dfc75640 10 monclient(hunting): authenticate will time out at 1333.871512s 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21de212640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21d8145e80 0x7f21d81d7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dd210640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21d8145a40 0x7f21d80aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21dda11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 0x7f21d80adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21de212640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21d8145e80 0x7f21d81d7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44866/0 (socket says 172.21.15.97:44866) 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21de212640 1 -- 172.21.15.97:0/4275992820 learned_addr learned my addr 172.21.15.97:0/4275992820 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21de212640 10 monclient(hunting): get_auth_request con 0x7f21d8145e80 auth_method 0 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21de212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:31.503 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21de212640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.493+0000 7f21de212640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dd210640 10 monclient(hunting): get_auth_request con 0x7f21d8145a40 auth_method 0 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dd210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dd210640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dd210640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dda11640 10 monclient(hunting): get_auth_request con 0x7f21d80a8a00 auth_method 0 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dda11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dda11640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dda11640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21de212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21de212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21de212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dd210640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:31.504 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dd210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dd210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dda11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dda11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.494+0000 7f21dda11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 1 -- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21d8145a40 msgr2=0x7f21d80aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21d8145a40 0x7f21d80aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 1 -- 172.21.15.97:0/4275992820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 msgr2=0x7f21d80adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 0x7f21d80adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 10 monclient: _finish_hunting 0 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 1 monclient: found mon.noname-a 2023-12-07T07:33:31.505 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:31.506 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:31.506 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.497+0000 7f21de212640 1 -- 172.21.15.97:0/4275992820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d4009400 con 0x7f21d8145e80 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.499+0000 7f21de212640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21d8145e80 0x7f21d81d7450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009d30 tx=0x7f21d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.499+0000 7f21ce7fc640 10 client.?.objecter ms_handle_connect 0x7f21d8145e80 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.499+0000 7f21ce7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21ce7fc640 1 -- 172.21.15.97:0/4275992820 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0009ba0 con 0x7f21d8145e80 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21ce7fc640 10 monclient: dump: 2023-12-07T07:33:31.507 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21ce7fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21ce7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:31.508 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 5 monclient: authenticate success, global_id 4227 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 10 monclient: _renew_subs 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d80ad060 con 0x7f21d8145e80 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 10 monclient: _renew_subs 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80b0890 con 0x7f21d8145e80 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.502+0000 7f21dfc75640 1 librados: init done 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.503+0000 7f21dfc75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.503+0000 7f21dfc75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.503+0000 7f21dfc75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:31.509 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.503+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d81d7170 con 0x7f21d8145e80 2023-12-07T07:33:31.717 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.713+0000 7f21ce7fc640 1 -- 172.21.15.97:0/4275992820 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d000f040 con 0x7f21d8145e80 2023-12-07T07:33:31.717 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.713+0000 7f21ce7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:31.717 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.713+0000 7f21cffff640 4 set_mon_vals no callback set 2023-12-07T07:33:31.756 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.754+0000 7f21ce7fc640 1 -- 172.21.15.97:0/4275992820 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0014680 con 0x7f21d8145e80 2023-12-07T07:33:31.756 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.754+0000 7f21ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:31.756 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.754+0000 7f21ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:31.756 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.754+0000 7f21ce7fc640 10 monclient: dump: 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:31.757 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:31.760 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.757+0000 7f21ce7fc640 1 -- 172.21.15.97:0/4275992820 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f21d00147e0 con 0x7f21d8145e80 2023-12-07T07:33:31.760 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.758+0000 7f21ce7fc640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f21bc06c740 0x7f21bc06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:31.760 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.758+0000 7f21dda11640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f21bc06c740 0x7f21bc06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:31.760 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.758+0000 7f21dda11640 10 monclient: get_auth_request con 0x7f21bc06c740 auth_method 0 2023-12-07T07:33:31.760 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.758+0000 7f21ce7fc640 10 client.4227.objecter ms_handle_connect 0x7f21bc06c740 2023-12-07T07:33:31.761 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.758+0000 7f21dda11640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f21bc06c740 0x7f21bc06ec00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f21d4036000 tx=0x7f21d403c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:31.773 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 1 -- 172.21.15.97:0/4275992820 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3678+0+0 (secure 0 0 0) 0x7f21d0059590 con 0x7f21d8145e80 2023-12-07T07:33:31.773 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 10 client.4227.objecter ms_dispatch 0x7f21d8000b90 osd_map(14..14 src has 1..14) v4 2023-12-07T07:33:31.773 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 3 client.4227.objecter handle_osd_map got epochs [14,14] > 0 2023-12-07T07:33:31.773 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 3 client.4227.objecter handle_osd_map decoding full epoch 14 2023-12-07T07:33:31.773 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 20 client.4227.objecter dump_active .. 0 homeless 2023-12-07T07:33:31.774 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 1 -- 172.21.15.97:0/4275992820 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21d00558d0 con 0x7f21d8145e80 2023-12-07T07:33:31.774 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:31.774 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:31.771+0000 7f21ce7fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:31.885 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-12-07T07:33:31.869+0000 1425b640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:32.002 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.000+0000 7f21dfc75640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:32.003 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.000+0000 7f21dfc75640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:32.003 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.000+0000 7f21dfc75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:32.003 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.000+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21d80b0890 con 0x7f21d8145e80 2023-12-07T07:33:32.094 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.092+0000 7f21ce7fc640 1 -- 172.21.15.97:0/4275992820 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10933 (secure 0 0 0) 0x7f21d80b0890 con 0x7f21d8145e80 2023-12-07T07:33:32.095 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.092+0000 7f21ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:32.095 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.092+0000 7f21ce7fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:32.095 INFO:teuthology.misc.health.smithi097.stdout: 2023-12-07T07:33:32.096 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":15,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:33:31.777283+0000","last_up_change":"2023-12-07T07:33:30.679434+0000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T07:33:30.063715+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6809","nonce":1781054720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6811","nonce":1781054720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6815","nonce":1781054720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6813","nonce":1781054720}]},"public_addr":"172.21.15.154:6809/1781054720","cluster_addr":"172.21.15.154:6811/1781054720","heartbeat_back_addr":"172.21.15.154:6815/1781054720","heartbeat_front_addr":"172.21.15.154:6813/1781054720","state":["exists","up"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6801","nonce":4174110211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6803","nonce":4174110211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6807","nonce":4174110211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6805","nonce":4174110211}]},"public_addr":"172.21.15.154:6801/4174110211","cluster_addr":"172.21.15.154:6803/4174110211","heartbeat_back_addr":"172.21.15.154:6807/4174110211","heartbeat_front_addr":"172.21.15.154:6805/4174110211","state":["exists","up"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6825","nonce":1615374996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6827","nonce":1615374996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6831","nonce":1615374996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6829","nonce":1615374996}]},"public_addr":"172.21.15.154:6825/1615374996","cluster_addr":"172.21.15.154:6827/1615374996","heartbeat_back_addr":"172.21.15.154:6831/1615374996","heartbeat_front_addr":"172.21.15.154:6829/1615374996","state":["exists","up"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6817","nonce":1161771449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6819","nonce":1161771449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6823","nonce":1161771449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6821","nonce":1161771449}]},"public_addr":"172.21.15.154:6817/1161771449","cluster_addr":"172.21.15.154:6819/1161771449","heartbeat_back_addr":"172.21.15.154:6823/1161771449","heartbeat_front_addr":"172.21.15.154:6821/1161771449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.804774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.252422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:28.225222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.956517+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 10 librados: watch_flush enter 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 10 librados: watch_flush exit 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 20 client.4227.objecter shutdown clearing up homeless session... 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 10 client.4227.objecter successfully canceled tick 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f21bc06c740 msgr2=0x7f21bc06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f21bc06c740 0x7f21bc06ec00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f21d4036000 tx=0x7f21d403c040 comp rx=0 tx=0).stop 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 10 monclient: shutdown 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21d8145e80 msgr2=0x7f21d81d7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21d8145e80 0x7f21d81d7450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009d30 tx=0x7f21d000c490 comp rx=0 tx=0).stop 2023-12-07T07:33:32.097 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 shutdown_connections 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f21bc06c740 0x7f21bc06ec00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21d8145e80 0x7f21d81d7450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21d8145a40 0x7f21d80aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 --2- 172.21.15.97:0/4275992820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21d80a8a00 0x7f21d80adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 >> 172.21.15.97:0/4275992820 conn(0x7f21d8066e80 msgr2=0x7f21d80b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 shutdown_connections 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 -- 172.21.15.97:0/4275992820 wait complete. 2023-12-07T07:33:32.098 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:32.094+0000 7f21dfc75640 1 librados: shutdown 2023-12-07T07:33:32.106 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-12-07T07:33:32.332 INFO:tasks.ceph.osd.3.smithi097.stderr:2023-12-07T07:33:32.327+0000 c493640 -1 osd.3 0 waiting for initial osdmap 2023-12-07T07:33:32.350 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-12-07T07:33:32.344+0000 ac7d640 -1 osd.1 0 waiting for initial osdmap 2023-12-07T07:33:32.400 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:33:32.394+0000 ac7d640 -1 osd.2 0 waiting for initial osdmap 2023-12-07T07:33:32.852 INFO:tasks.ceph.osd.3.smithi097.stderr:2023-12-07T07:33:32.837+0000 13a5a640 -1 osd.3 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:32.875 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:33:32.859+0000 13a5a640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:32.885 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-12-07T07:33:32.869+0000 13259640 -1 osd.1 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T07:33:38.913 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:39.003 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.001+0000 7fc2ea24b640 10 monclient: get_monmap_and_config 2023-12-07T07:33:39.003 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.001+0000 7fc2ea24b640 10 monclient: build_initial_monmap 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.001+0000 7fc2ea24b640 10 monclient: monmap: 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:33:39.002660+0000 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:33:39.002660+0000 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:39.004 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.002+0000 7fc2ea24b640 1 Processor -- start 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.002+0000 7fc2ea24b640 1 -- start start 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.002+0000 7fc2ea24b640 10 monclient: init 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.002+0000 7fc2ea24b640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.002+0000 7fc2ea24b640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc2e40a8a00 0x7fc2e40a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e40a8a00 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc2e40b2260 0x7fc2e41425b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e40b2260 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2e4142af0 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e40aa970 con 0x7fc2e4142af0 2023-12-07T07:33:39.006 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e40aac50 con 0x7fc2e40b2260 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e40aaf30 con 0x7fc2e40a8a00 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2ea24b640 10 monclient(hunting): authenticate will time out at 1341.381504s 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2e2ffd640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc2e40b2260 0x7fc2e41425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.003+0000 7fc2e37fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc2e40a8a00 0x7fc2e40a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc2e40b2260 0x7fc2e41425b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44900/0 (socket says 172.21.15.97:44900) 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 1 -- 172.21.15.97:0/460873933 learned_addr learned my addr 172.21.15.97:0/460873933 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc2e40b2260 auth_method 0 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.007 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.008 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e37fe640 10 monclient(hunting): get_auth_request con 0x7fc2e40a8a00 auth_method 0 2023-12-07T07:33:39.008 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.008 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.008 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.008 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.008 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.009 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.004+0000 7fc2e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.009 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.005+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.009 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.005+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.009 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.005+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.009 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2023-12-07T07:33:39.009 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 1 -- 172.21.15.97:0/460873933 >> v2:172.21.15.97:3300/0 conn(0x7fc2e40b2260 msgr2=0x7fc2e41425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:39.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 1 --2- 172.21.15.97:0/460873933 >> v2:172.21.15.97:3300/0 conn(0x7fc2e40b2260 0x7fc2e41425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 1 -- 172.21.15.97:0/460873933 >> v1:172.21.15.154:6789/0 conn(0x7fc2e4142af0 legacy=0x7fc2e413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:39.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 1 monclient: found mon.noname-c 2023-12-07T07:33:39.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:39.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.010 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.007+0000 7fc2e37fe640 1 -- 172.21.15.97:0/460873933 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e4142090 con 0x7fc2e40a8a00 2023-12-07T07:33:39.013 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.011+0000 7fc2e37fe640 1 --2- 172.21.15.97:0/460873933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e40a8de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40b5370 tx=0x7fc2d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce61f33c47cba92d server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 1 -- 172.21.15.97:0/460873933 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4031830 con 0x7fc2e40a8a00 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: dump: 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.018 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 1 -- 172.21.15.97:0/460873933 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d4031990 con 0x7fc2e40a8a00 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 5 monclient: authenticate success, global_id 4270 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 1 -- 172.21.15.97:0/460873933 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4031d20 con 0x7fc2e40a8a00 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 4 set_mon_vals no callback set 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2e27fc640 10 monclient: dump: 2023-12-07T07:33:39.019 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 10 monclient: shutdown 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 -- 172.21.15.97:0/460873933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 msgr2=0x7fc2e40a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/460873933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e40a8de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40b5370 tx=0x7fc2d402b8d0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 -- 172.21.15.97:0/460873933 shutdown_connections 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/460873933 >> v2:172.21.15.97:3300/0 conn(0x7fc2e40b2260 0x7fc2e41425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.020 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/460873933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e40a8de0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.021 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 -- 172.21.15.97:0/460873933 >> 172.21.15.97:0/460873933 conn(0x7fc2e4066ea0 msgr2=0x7fc2e40b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:39.021 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 -- 172.21.15.97:0/460873933 shutdown_connections 2023-12-07T07:33:39.021 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.015+0000 7fc2ea24b640 1 -- 172.21.15.97:0/460873933 wait complete. 2023-12-07T07:33:39.023 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.016+0000 7fc2ea24b640 10 monclient: build_initial_monmap 2023-12-07T07:33:39.023 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.016+0000 7fc2ea24b640 10 monclient: monmap: 2023-12-07T07:33:39.023 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2023-12-07T07:33:39.023 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.016+0000 7fc2ea24b640 1 librados: starting msgr at 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.016+0000 7fc2ea24b640 1 librados: starting objecter 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.016+0000 7fc2ea24b640 1 Processor -- start 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.017+0000 7fc2ea24b640 1 -- start start 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.017+0000 7fc2ea24b640 1 librados: setting wanted keys 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.017+0000 7fc2ea24b640 1 librados: calling monclient init 2023-12-07T07:33:39.024 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.017+0000 7fc2ea24b640 10 monclient: init 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e4136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e40a8a00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2e40b2260 0x7fc2e4145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e40b2260 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e4142af0 0x7fc2e41d7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e4142af0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e41d6ea0 con 0x7fc2e4142af0 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e4144f30 con 0x7fc2e40a8a00 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e4146290 con 0x7fc2e40b2260 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:39.025 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2ea24b640 10 monclient(hunting): authenticate will time out at 1341.396425s 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e4136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e3fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e4142af0 0x7fc2e41d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e2ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2e40b2260 0x7fc2e4145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e4136020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:33396/0 (socket says 172.21.15.97:33396) 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e37fe640 1 -- 172.21.15.97:0/611573357 learned_addr learned my addr 172.21.15.97:0/611573357 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e37fe640 10 monclient(hunting): get_auth_request con 0x7fc2e40a8a00 auth_method 0 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.018+0000 7fc2e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e3fff640 10 monclient(hunting): get_auth_request con 0x7fc2e4142af0 auth_method 0 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.026 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc2e40b2260 auth_method 0 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.027 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.019+0000 7fc2e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 1 -- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2e40b2260 msgr2=0x7fc2e4145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2e40b2260 0x7fc2e4145d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 1 -- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e4142af0 msgr2=0x7fc2e41d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e4142af0 0x7fc2e41d7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 1 monclient: found mon.noname-b 2023-12-07T07:33:39.028 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:39.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.022+0000 7fc2e37fe640 1 -- 172.21.15.97:0/611573357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d4009400 con 0x7fc2e40a8a00 2023-12-07T07:33:39.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.024+0000 7fc2e37fe640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e4136020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009890 tx=0x7fc2d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:39.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.024+0000 7fc2cbfff640 10 client.?.objecter ms_handle_connect 0x7fc2e40a8a00 2023-12-07T07:33:39.029 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.024+0000 7fc2cbfff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:39.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 1 -- 172.21.15.97:0/611573357 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d403b970 con 0x7fc2e40a8a00 2023-12-07T07:33:39.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:39.030 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: dump: 2023-12-07T07:33:39.030 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:39.030 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.030 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: _finish_auth 0 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 1 -- 172.21.15.97:0/611573357 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d403bad0 con 0x7fc2e40a8a00 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 5 monclient: authenticate success, global_id 4273 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 10 monclient: _renew_subs 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.031 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e4137a30 con 0x7fc2e40a8a00 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 1 -- 172.21.15.97:0/611573357 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d403be80 con 0x7fc2e40a8a00 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2cbfff640 10 monclient: dump: 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr: 2023-12-07T07:33:39.032 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2e1ffb640 4 set_mon_vals no callback set 2023-12-07T07:33:39.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:39.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 10 monclient: _renew_subs 2023-12-07T07:33:39.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e41d7a50 con 0x7fc2e40a8a00 2023-12-07T07:33:39.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.027+0000 7fc2ea24b640 1 librados: init done 2023-12-07T07:33:39.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.028+0000 7fc2ea24b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:39.033 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.029+0000 7fc2ea24b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:39.035 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.029+0000 7fc2ea24b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.036 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.029+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e4137740 con 0x7fc2e40a8a00 2023-12-07T07:33:39.277 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.275+0000 7fc2cbfff640 1 -- 172.21.15.97:0/611573357 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fc2d40453f0 con 0x7fc2e40a8a00 2023-12-07T07:33:39.277 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.275+0000 7fc2cbfff640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2d006c790 0x7fc2d006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.278 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.275+0000 7fc2e2ffd640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2d006c790 0x7fc2d006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.278 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.275+0000 7fc2e2ffd640 10 monclient: get_auth_request con 0x7fc2d006c790 auth_method 0 2023-12-07T07:33:39.278 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.276+0000 7fc2e2ffd640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2d006c790 0x7fc2d006ec50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4145a30 tx=0x7fc2d8009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:39.278 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.276+0000 7fc2cbfff640 10 client.4273.objecter ms_handle_connect 0x7fc2d006c790 2023-12-07T07:33:39.279 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.277+0000 7fc2cbfff640 1 -- 172.21.15.97:0/611573357 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc2d407ab10 con 0x7fc2e40a8a00 2023-12-07T07:33:39.279 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.277+0000 7fc2cbfff640 10 client.4273.objecter ms_dispatch 0x7fc2e4000b90 osd_map(20..20 src has 1..20) v4 2023-12-07T07:33:39.280 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.277+0000 7fc2cbfff640 3 client.4273.objecter handle_osd_map got epochs [20,20] > 0 2023-12-07T07:33:39.280 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.277+0000 7fc2cbfff640 3 client.4273.objecter handle_osd_map decoding full epoch 20 2023-12-07T07:33:39.280 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.277+0000 7fc2cbfff640 20 client.4273.objecter dump_active .. 0 homeless 2023-12-07T07:33:39.289 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.287+0000 7fc2cbfff640 1 -- 172.21.15.97:0/611573357 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2d4076e50 con 0x7fc2e40a8a00 2023-12-07T07:33:39.289 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.287+0000 7fc2cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:39.290 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.287+0000 7fc2cbfff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:39.462 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.460+0000 7fc2ea24b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:39.463 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.460+0000 7fc2ea24b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:39.463 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.460+0000 7fc2ea24b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.463 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.460+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2e4069c00 con 0x7fc2e40a8a00 2023-12-07T07:33:39.550 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.548+0000 7fc2cbfff640 1 -- 172.21.15.97:0/611573357 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+12840 (secure 0 0 0) 0x7fc2d4035090 con 0x7fc2e40a8a00 2023-12-07T07:33:39.550 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.548+0000 7fc2cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:39.550 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.548+0000 7fc2cbfff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:39.550 INFO:teuthology.misc.health.smithi097.stdout: 2023-12-07T07:33:39.551 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":20,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:33:37.456598+0000","last_up_change":"2023-12-07T07:33:33.953336+0000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T07:33:30.063715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6819","nonce":2798215727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6821","nonce":2798215727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6825","nonce":2798215727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6823","nonce":2798215727}]},"public_addr":"172.21.15.97:6819/2798215727","cluster_addr":"172.21.15.97:6821/2798215727","heartbeat_back_addr":"172.21.15.97:6825/2798215727","heartbeat_front_addr":"172.21.15.97:6823/2798215727","state":["exists","up"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6827","nonce":138440183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6829","nonce":138440183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6833","nonce":138440183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6831","nonce":138440183}]},"public_addr":"172.21.15.97:6827/138440183","cluster_addr":"172.21.15.97:6829/138440183","heartbeat_back_addr":"172.21.15.97:6833/138440183","heartbeat_front_addr":"172.21.15.97:6831/138440183","state":["exists","up"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6803","nonce":1673674356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6805","nonce":1673674356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6809","nonce":1673674356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6807","nonce":1673674356}]},"public_addr":"172.21.15.97:6803/1673674356","cluster_addr":"172.21.15.97:6805/1673674356","heartbeat_back_addr":"172.21.15.97:6809/1673674356","heartbeat_front_addr":"172.21.15.97:6807/1673674356","state":["exists","up"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6811","nonce":929626392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6813","nonce":929626392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6817","nonce":929626392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6815","nonce":929626392}]},"public_addr":"172.21.15.97:6811/929626392","cluster_addr":"172.21.15.97:6813/929626392","heartbeat_back_addr":"172.21.15.97:6817/929626392","heartbeat_front_addr":"172.21.15.97:6815/929626392","state":["exists","up"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6809","nonce":1781054720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6811","nonce":1781054720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6815","nonce":1781054720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6813","nonce":1781054720}]},"public_addr":"172.21.15.154:6809/1781054720","cluster_addr":"172.21.15.154:6811/1781054720","heartbeat_back_addr":"172.21.15.154:6815/1781054720","heartbeat_front_addr":"172.21.15.154:6813/1781054720","state":["exists","up"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6801","nonce":4174110211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6803","nonce":4174110211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6807","nonce":4174110211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6805","nonce":4174110211}]},"public_addr":"172.21.15.154:6801/4174110211","cluster_addr":"172.21.15.154:6803/4174110211","heartbeat_back_addr":"172.21.15.154:6807/4174110211","heartbeat_front_addr":"172.21.15.154:6805/4174110211","state":["exists","up"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6825","nonce":1615374996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6827","nonce":1615374996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6831","nonce":1615374996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6829","nonce":1615374996}]},"public_addr":"172.21.15.154:6825/1615374996","cluster_addr":"172.21.15.154:6827/1615374996","heartbeat_back_addr":"172.21.15.154:6831/1615374996","heartbeat_front_addr":"172.21.15.154:6829/1615374996","state":["exists","up"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6817","nonce":1161771449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6819","nonce":1161771449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6823","nonce":1161771449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6821","nonce":1161771449}]},"public_addr":"172.21.15.154:6817/1161771449","cluster_addr":"172.21.15.154:6819/1161771449","heartbeat_back_addr":"172.21.15.154:6823/1161771449","heartbeat_front_addr":"172.21.15.154:6821/1161771449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.927364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.534718+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.623805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.804774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.252422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:28.225222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.956517+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 10 librados: watch_flush enter 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 10 librados: watch_flush exit 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 20 client.4273.objecter shutdown clearing up homeless session... 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 10 client.4273.objecter successfully canceled tick 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2d006c790 msgr2=0x7fc2d006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2d006c790 0x7fc2d006ec50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4145a30 tx=0x7fc2d8009240 comp rx=0 tx=0).stop 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 10 monclient: shutdown 2023-12-07T07:33:39.553 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 msgr2=0x7fc2e4136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e4136020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009890 tx=0x7fc2d40046c0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 shutdown_connections 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2d006c790 0x7fc2d006ec50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2e4142af0 0x7fc2e41d7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2e40b2260 0x7fc2e4145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 --2- 172.21.15.97:0/611573357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40a8a00 0x7fc2e4136020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 >> 172.21.15.97:0/611573357 conn(0x7fc2e4066ea0 msgr2=0x7fc2e413eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 shutdown_connections 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 -- 172.21.15.97:0/611573357 wait complete. 2023-12-07T07:33:39.554 INFO:teuthology.misc.health.smithi097.stderr:2023-12-07T07:33:39.550+0000 7fc2ea24b640 1 librados: shutdown 2023-12-07T07:33:39.563 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-07T07:33:39.563 INFO:tasks.ceph:Creating RBD pool 2023-12-07T07:33:39.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.685+0000 7f8e45675640 10 monclient: get_monmap_and_config 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.685+0000 7f8e45675640 10 monclient: build_initial_monmap 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.685+0000 7f8e45675640 10 monclient: monmap: 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:39.686165+0000 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:39.686165+0000 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:39.687 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:39.688 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:39.688 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.685+0000 7f8e45675640 1 Processor -- start 2023-12-07T07:33:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.685+0000 7f8e45675640 1 -- start start 2023-12-07T07:33:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.685+0000 7f8e45675640 10 monclient: init 2023-12-07T07:33:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-12-07T07:33:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e400a8a70 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8e400a8e50 0x7f8e40142890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient(hunting): picked mon.noname-a con 0x7f8e400a8e50 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8e4013e9c0 0x7f8e4013ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4013e9c0 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400a9e70 con 0x7f8e400a8a70 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e40142dd0 con 0x7f8e400a8e50 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e400aa9e0 con 0x7f8e4013e9c0 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.686+0000 7f8e45675640 10 monclient(hunting): authenticate will time out at 1342.064982s 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.687+0000 7f8e3effd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8e400a8a70 0x7f8e40142180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39570/0 (socket says 172.21.15.97:39570) 2023-12-07T07:33:39.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.687+0000 7f8e3effd640 1 -- 172.21.15.97:0/1577831539 learned_addr learned my addr 172.21.15.97:0/1577831539 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.688+0000 7f8e3e7fc640 1 --2- 172.21.15.97:0/1577831539 >> v2:172.21.15.97:3300/0 conn(0x7f8e400a8e50 0x7f8e40142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.689+0000 7f8e3e7fc640 10 monclient(hunting): get_auth_request con 0x7f8e400a8e50 auth_method 0 2023-12-07T07:33:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.689+0000 7f8e3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.689+0000 7f8e3e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.689+0000 7f8e3e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.690+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.690+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.690+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.693+0000 7f8e3f7fe640 1 --2- 172.21.15.97:0/1577831539 >> v2:172.21.15.154:3300/0 conn(0x7f8e4013e9c0 0x7f8e4013ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 1 -- 172.21.15.97:0/1577831539 >> v2:172.21.15.154:3300/0 conn(0x7f8e4013e9c0 msgr2=0x7f8e4013ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 1 --2- 172.21.15.97:0/1577831539 >> v2:172.21.15.154:3300/0 conn(0x7f8e4013e9c0 0x7f8e4013ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 1 -- 172.21.15.97:0/1577831539 >> v1:172.21.15.97:6789/0 conn(0x7f8e400a8a70 legacy=0x7f8e40142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 1 monclient: found mon.noname-a 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:39.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:39.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.696+0000 7f8e3e7fc640 1 -- 172.21.15.97:0/1577831539 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e400aacc0 con 0x7f8e400a8e50 2023-12-07T07:33:39.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.698+0000 7f8e3e7fc640 1 --2- 172.21.15.97:0/1577831539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e400a8e50 0x7f8e40142890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8e400aa0a0 tx=0x7f8e3402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f37765367975d7f server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:39.707 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 1 -- 172.21.15.97:0/1577831539 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3402e070 con 0x7f8e400a8e50 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: dump: 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 1 -- 172.21.15.97:0/1577831539 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e340315a0 con 0x7f8e400a8e50 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e45675640 5 monclient: authenticate success, global_id 4242 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e45675640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e45675640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e45675640 4 set_mon_vals no callback set 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 1 -- 172.21.15.97:0/1577831539 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34031930 con 0x7f8e400a8e50 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.705+0000 7f8e3dffb640 10 monclient: dump: 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.710 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:39.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 10 monclient: shutdown 2023-12-07T07:33:39.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:39.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 -- 172.21.15.97:0/1577831539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e400a8e50 msgr2=0x7f8e40142890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 --2- 172.21.15.97:0/1577831539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e400a8e50 0x7f8e40142890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8e400aa0a0 tx=0x7f8e3402b840 comp rx=0 tx=0).stop 2023-12-07T07:33:39.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 -- 172.21.15.97:0/1577831539 shutdown_connections 2023-12-07T07:33:39.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 --2- 172.21.15.97:0/1577831539 >> v2:172.21.15.154:3300/0 conn(0x7f8e4013e9c0 0x7f8e4013ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 --2- 172.21.15.97:0/1577831539 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e400a8e50 0x7f8e40142890 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 -- 172.21.15.97:0/1577831539 >> 172.21.15.97:0/1577831539 conn(0x7f8e40061a90 msgr2=0x7f8e400b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 -- 172.21.15.97:0/1577831539 shutdown_connections 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.706+0000 7f8e45675640 1 -- 172.21.15.97:0/1577831539 wait complete. 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.707+0000 7f8e45675640 10 monclient: build_initial_monmap 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.707+0000 7f8e45675640 10 monclient: monmap: 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.708+0000 7f8e45675640 1 librados: starting msgr at 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.708+0000 7f8e45675640 1 librados: starting objecter 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.708+0000 7f8e45675640 1 Processor -- start 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.708+0000 7f8e45675640 1 -- start start 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.708+0000 7f8e45675640 1 librados: setting wanted keys 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.708+0000 7f8e45675640 1 librados: calling monclient init 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.708+0000 7f8e45675640 10 monclient: init 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8e400a8a70 0x7f8e4013a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): picked mon.noname-c con 0x7f8e400a8a70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e400a8e50 0x7f8e4013af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): picked mon.noname-b con 0x7f8e400a8e50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e4013e9c0 0x7f8e401dae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4013e9c0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e400aee20 con 0x7f8e4013e9c0 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4013b470 con 0x7f8e400a8e50 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e401c8ba0 con 0x7f8e400a8a70 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.709+0000 7f8e45675640 10 monclient(hunting): authenticate will time out at 1342.087593s 2023-12-07T07:33:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e400a8e50 0x7f8e4013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e400a8e50 0x7f8e4013af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:33414/0 (socket says 172.21.15.97:33414) 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 1 -- 172.21.15.97:0/2296276065 learned_addr learned my addr 172.21.15.97:0/2296276065 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 10 monclient(hunting): get_auth_request con 0x7f8e400a8e50 auth_method 0 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3effd640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8e400a8a70 0x7f8e4013a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3effd640 10 monclient(hunting): get_auth_request con 0x7f8e400a8a70 auth_method 0 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:39.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3effd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:39.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.710+0000 7f8e3effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:39.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.711+0000 7f8e3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:39.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.711+0000 7f8e3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:39.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.711+0000 7f8e3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:39.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 1 -- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8e400a8a70 msgr2=0x7f8e4013a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8e400a8a70 0x7f8e4013a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 1 -- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e4013e9c0 msgr2=0x7f8e401dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e4013e9c0 0x7f8e401dae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 1 monclient: found mon.noname-b 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.713+0000 7f8e3e7fc640 1 -- 172.21.15.97:0/2296276065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e34009400 con 0x7f8e400a8e50 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.716+0000 7f8e3e7fc640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e400a8e50 0x7f8e4013af30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8e34000c00 tx=0x7f8e340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.716+0000 7f8e277fe640 10 client.?.objecter ms_handle_connect 0x7f8e400a8e50 2023-12-07T07:33:39.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.716+0000 7f8e277fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:39.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34032900 con 0x7f8e400a8e50 2023-12-07T07:33:39.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: dump: 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e34032a60 con 0x7f8e400a8e50 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 5 monclient: authenticate success, global_id 4276 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34032dd0 con 0x7f8e400a8e50 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e277fe640 10 monclient: dump: 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:39.723 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 10 monclient: _renew_subs 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e400b0bd0 con 0x7f8e400a8e50 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e3d7fa640 4 set_mon_vals no callback set 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 10 monclient: _renew_subs 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e401c8830 con 0x7f8e400a8e50 2023-12-07T07:33:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.719+0000 7f8e45675640 1 librados: init done 2023-12-07T07:33:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.720+0000 7f8e45675640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.720+0000 7f8e45675640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:39.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.720+0000 7f8e45675640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:39.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.720+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e40067ac0 con 0x7f8e400a8e50 2023-12-07T07:33:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.971+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f8e3404d3f0 con 0x7f8e400a8e50 2023-12-07T07:33:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.971+0000 7f8e277fe640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8e2c06c790 0x7f8e2c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.971+0000 7f8e3effd640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8e2c06c790 0x7f8e2c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.971+0000 7f8e3effd640 10 monclient: get_auth_request con 0x7f8e2c06c790 auth_method 0 2023-12-07T07:33:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.972+0000 7f8e277fe640 10 client.4276.objecter ms_handle_connect 0x7f8e2c06c790 2023-12-07T07:33:39.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.972+0000 7f8e3effd640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8e2c06c790 0x7f8e2c06ec50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8e30005880 tx=0x7f8e30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.974+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e3407ac40 con 0x7f8e400a8e50 2023-12-07T07:33:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.974+0000 7f8e277fe640 10 client.4276.objecter ms_dispatch 0x7f8e40000b90 osd_map(20..20 src has 1..20) v4 2023-12-07T07:33:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.974+0000 7f8e277fe640 3 client.4276.objecter handle_osd_map got epochs [20,20] > 0 2023-12-07T07:33:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.974+0000 7f8e277fe640 3 client.4276.objecter handle_osd_map decoding full epoch 20 2023-12-07T07:33:39.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.974+0000 7f8e277fe640 20 client.4276.objecter dump_active .. 0 homeless 2023-12-07T07:33:39.986 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.984+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e34076880 con 0x7f8e400a8e50 2023-12-07T07:33:39.986 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.984+0000 7f8e277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:39.986 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:39.984+0000 7f8e277fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:40.132 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.130+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8e34076c50 con 0x7f8e400a8e50 2023-12-07T07:33:40.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.177+0000 7f8e45675640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-07T07:33:40.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.177+0000 7f8e45675640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-07T07:33:40.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.177+0000 7f8e45675640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:40.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.177+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f8e4013ee30 con 0x7f8e400a8e50 2023-12-07T07:33:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.909+0000 7f8e277fe640 1 -- 172.21.15.97:0/2296276065 <== mon.1 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v21) v1 ==== 109+0+0 (secure 0 0 0) 0x7f8e340a8830 con 0x7f8e400a8e50 2023-12-07T07:33:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.909+0000 7f8e277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-07T07:33:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.909+0000 7f8e277fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-12-07T07:33:40.912 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'rbd' created 2023-12-07T07:33:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.911+0000 7f8e45675640 10 librados: watch_flush enter 2023-12-07T07:33:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.911+0000 7f8e45675640 10 librados: watch_flush exit 2023-12-07T07:33:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.911+0000 7f8e45675640 20 client.4276.objecter shutdown clearing up homeless session... 2023-12-07T07:33:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.911+0000 7f8e45675640 10 client.4276.objecter successfully canceled tick 2023-12-07T07:33:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8e2c06c790 msgr2=0x7f8e2c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8e2c06c790 0x7f8e2c06ec50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8e30005880 tx=0x7f8e30009040 comp rx=0 tx=0).stop 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 10 monclient: shutdown 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e400a8e50 msgr2=0x7f8e4013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e400a8e50 0x7f8e4013af30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8e34000c00 tx=0x7f8e340046c0 comp rx=0 tx=0).stop 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 shutdown_connections 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f8e2c06c790 0x7f8e2c06ec50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e4013e9c0 0x7f8e401dae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e400a8e50 0x7f8e4013af30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 --2- 172.21.15.97:0/2296276065 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8e400a8a70 0x7f8e4013a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 >> 172.21.15.97:0/2296276065 conn(0x7f8e40061a90 msgr2=0x7f8e40142180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 shutdown_connections 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 -- 172.21.15.97:0/2296276065 wait complete. 2023-12-07T07:33:40.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:40.912+0000 7f8e45675640 1 librados: shutdown 2023-12-07T07:33:40.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-07T07:33:41.034 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.032+0000 7fe6bae10640 10 monclient: get_monmap_and_config 2023-12-07T07:33:41.034 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.032+0000 7fe6bae10640 10 monclient: build_initial_monmap 2023-12-07T07:33:41.034 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.032+0000 7fe6bae10640 10 monclient: monmap: 2023-12-07T07:33:41.034 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:41.034 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:41.034 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:41.033268+0000 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:41.033268+0000 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:41.035 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.032+0000 7fe6bae10640 1 Processor -- start 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.032+0000 7fe6bae10640 1 -- start start 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.032+0000 7fe6bae10640 10 monclient: init 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6b4145840 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4146080 0x7fe6b413f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:41.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient(hunting): picked mon.noname-b con 0x7fe6b4146080 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe6b413fa50 0x7fe6b414c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b413fa50 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b40aa850 con 0x7fe6b4145840 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b4145c20 con 0x7fe6b413fa50 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40b2800 con 0x7fe6b4146080 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.033+0000 7fe6bae10640 10 monclient(hunting): authenticate will time out at 1343.411903s 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4146080 0x7fe6b413f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b93ad640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe6b413fa50 0x7fe6b414c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b8bac640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe6b4145840 0x7fe6b40b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44656/0 (socket says 172.21.15.97:44656) 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b8bac640 1 -- 172.21.15.97:0/2265903541 learned_addr learned my addr 172.21.15.97:0/2265903541 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:41.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 10 monclient(hunting): get_auth_request con 0x7fe6b4146080 auth_method 0 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b93ad640 10 monclient(hunting): get_auth_request con 0x7fe6b413fa50 auth_method 0 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b93ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b93ad640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b93ad640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b93ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:41.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.034+0000 7fe6b93ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.035+0000 7fe6b93ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 1 -- 172.21.15.97:0/2265903541 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4146080 msgr2=0x7fe6b413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 1 --2- 172.21.15.97:0/2265903541 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4146080 0x7fe6b413f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 1 -- 172.21.15.97:0/2265903541 >> v1:172.21.15.154:6789/0 conn(0x7fe6b4145840 legacy=0x7fe6b40b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 10 monclient: _finish_hunting 0 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 1 monclient: found mon.noname-a 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.036+0000 7fe6b93ad640 1 -- 172.21.15.97:0/2265903541 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40a9260 con 0x7fe6b413fa50 2023-12-07T07:33:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.038+0000 7fe6b93ad640 1 --2- 172.21.15.97:0/2265903541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b413fa50 0x7fe6b414c960 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009dc0 tx=0x7fe6ac02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13384497599513c server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 1 -- 172.21.15.97:0/2265903541 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac02cbb0 con 0x7fe6b413fa50 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: dump: 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:41.045 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 1 -- 172.21.15.97:0/2265903541 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ac02cd10 con 0x7fe6b413fa50 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 5 monclient: authenticate success, global_id 4248 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 1 -- 172.21.15.97:0/2265903541 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac034660 con 0x7fe6b413fa50 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 4 set_mon_vals no callback set 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6b37fe640 10 monclient: dump: 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 10 monclient: shutdown 2023-12-07T07:33:41.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 1 -- 172.21.15.97:0/2265903541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b413fa50 msgr2=0x7fe6b414c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.042+0000 7fe6bae10640 1 --2- 172.21.15.97:0/2265903541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b413fa50 0x7fe6b414c960 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009dc0 tx=0x7fe6ac02be00 comp rx=0 tx=0).stop 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 -- 172.21.15.97:0/2265903541 shutdown_connections 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 --2- 172.21.15.97:0/2265903541 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b413fa50 0x7fe6b414c960 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 --2- 172.21.15.97:0/2265903541 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4146080 0x7fe6b413f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 -- 172.21.15.97:0/2265903541 >> 172.21.15.97:0/2265903541 conn(0x7fe6b40b41f0 msgr2=0x7fe6b40b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 -- 172.21.15.97:0/2265903541 shutdown_connections 2023-12-07T07:33:41.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 -- 172.21.15.97:0/2265903541 wait complete. 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 10 monclient: build_initial_monmap 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 10 monclient: monmap: 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 librados: starting msgr at 2023-12-07T07:33:41.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.043+0000 7fe6bae10640 1 librados: starting objecter 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 Processor -- start 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 -- start start 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 librados: setting wanted keys 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 librados: calling monclient init 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient: init 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6b413fa50 0x7fe6b41dafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): picked mon.noname-b con 0x7fe6b413fa50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4145840 0x7fe6b41cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b4145840 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b4146080 0x7fe6b41cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:41.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): picked mon.noname-a con 0x7fe6b4146080 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41db670 con 0x7fe6b4146080 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41cd340 con 0x7fe6b413fa50 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41dbb00 con 0x7fe6b4145840 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.044+0000 7fe6bae10640 10 monclient(hunting): authenticate will time out at 1343.423165s 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b8bac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6b413fa50 0x7fe6b41dafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b93ad640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b4146080 0x7fe6b41cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b3fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4145840 0x7fe6b41cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b8bac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6b413fa50 0x7fe6b41dafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38390/0 (socket says 172.21.15.97:38390) 2023-12-07T07:33:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b8bac640 1 -- 172.21.15.97:0/1918466796 learned_addr learned my addr 172.21.15.97:0/1918466796 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b8bac640 10 monclient(hunting): get_auth_request con 0x7fe6b413fa50 auth_method 0 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b8bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b8bac640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b8bac640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b93ad640 10 monclient(hunting): get_auth_request con 0x7fe6b4146080 auth_method 0 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b93ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b93ad640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b93ad640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b3fff640 10 monclient(hunting): get_auth_request con 0x7fe6b4145840 auth_method 0 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.045+0000 7fe6b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b8bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b8bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b8bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b93ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b93ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.046+0000 7fe6b93ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.048+0000 7fe6b8bac640 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.048+0000 7fe6b8bac640 1 -- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4145840 msgr2=0x7fe6b41cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.048+0000 7fe6b8bac640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4145840 0x7fe6b41cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.048+0000 7fe6b8bac640 1 -- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b4146080 msgr2=0x7fe6b41cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.048+0000 7fe6b8bac640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b4146080 0x7fe6b41cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.049+0000 7fe6b8bac640 10 monclient: _finish_hunting 0 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.049+0000 7fe6b8bac640 1 monclient: found mon.noname-b 2023-12-07T07:33:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.049+0000 7fe6b8bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:41.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.049+0000 7fe6b8bac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:41.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.049+0000 7fe6b8bac640 1 -- 172.21.15.97:0/1918466796 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac009a70 con 0x7fe6b413fa50 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.051+0000 7fe6b8bac640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6b413fa50 0x7fe6b41dafe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800c4a0 tx=0x7fe6a800c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.051+0000 7fe6b17fa640 10 client.?.objecter ms_handle_connect 0x7fe6b413fa50 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.051+0000 7fe6b17fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 1 -- 172.21.15.97:0/1918466796 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8005ce0 con 0x7fe6b413fa50 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: dump: 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:41.058 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: _finish_auth 0 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: _check_auth_tickets 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 1 -- 172.21.15.97:0/1918466796 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a8005e40 con 0x7fe6b413fa50 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 1 -- 172.21.15.97:0/1918466796 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8007270 con 0x7fe6b413fa50 2023-12-07T07:33:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6bae10640 5 monclient: authenticate success, global_id 4282 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6bae10640 10 monclient: _renew_subs 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6bae10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b40b0d70 con 0x7fe6b413fa50 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b2ffd640 4 set_mon_vals no callback set 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.054+0000 7fe6b17fa640 10 monclient: dump: 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:41.060 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 10 monclient: _renew_subs 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b40b14a0 con 0x7fe6b413fa50 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 1 librados: init done 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:41.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.055+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b40b1090 con 0x7fe6b413fa50 2023-12-07T07:33:41.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.074+0000 7fe6b17fa640 1 -- 172.21.15.97:0/1918466796 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe6a8007490 con 0x7fe6b413fa50 2023-12-07T07:33:41.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.074+0000 7fe6b17fa640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fe69806c7d0 0x7fe69806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:41.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.074+0000 7fe6b3fff640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fe69806c7d0 0x7fe69806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:41.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.075+0000 7fe6b3fff640 10 monclient: get_auth_request con 0x7fe69806c7d0 auth_method 0 2023-12-07T07:33:41.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.075+0000 7fe6b17fa640 10 client.4282.objecter ms_handle_connect 0x7fe69806c7d0 2023-12-07T07:33:41.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.075+0000 7fe6b3fff640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fe69806c7d0 0x7fe69806ec90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40b3460 tx=0x7fe6a0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.320+0000 7fe6b17fa640 1 -- 172.21.15.97:0/1918466796 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5199+0+0 (secure 0 0 0) 0x7fe6a8057d60 con 0x7fe6b413fa50 2023-12-07T07:33:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.320+0000 7fe6b17fa640 10 client.4282.objecter ms_dispatch 0x7fe6b4000b90 osd_map(21..21 src has 1..21) v4 2023-12-07T07:33:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.320+0000 7fe6b17fa640 3 client.4282.objecter handle_osd_map got epochs [21,21] > 0 2023-12-07T07:33:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.320+0000 7fe6b17fa640 3 client.4282.objecter handle_osd_map decoding full epoch 21 2023-12-07T07:33:41.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.321+0000 7fe6b17fa640 20 client.4282.objecter dump_active .. 0 homeless 2023-12-07T07:33:41.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.331+0000 7fe6b17fa640 1 -- 172.21.15.97:0/1918466796 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6a80540a0 con 0x7fe6b413fa50 2023-12-07T07:33:41.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.331+0000 7fe6b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:41.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.331+0000 7fe6b17fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:41.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.509+0000 7fe6bae10640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-07T07:33:41.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.509+0000 7fe6bae10640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-07T07:33:41.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.509+0000 7fe6bae10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:41.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:41.509+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe6b4067a80 con 0x7fe6b413fa50 2023-12-07T07:33:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.004+0000 7fe6b17fa640 1 -- 172.21.15.97:0/1918466796 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v22) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe6a80170c0 con 0x7fe6b413fa50 2023-12-07T07:33:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.004+0000 7fe6b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-07T07:33:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.004+0000 7fe6b17fa640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-12-07T07:33:42.006 INFO:teuthology.orchestra.run.smithi097.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-07T07:33:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 10 librados: watch_flush enter 2023-12-07T07:33:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 10 librados: watch_flush exit 2023-12-07T07:33:42.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 20 client.4282.objecter shutdown clearing up homeless session... 2023-12-07T07:33:42.010 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 10 client.4282.objecter successfully canceled tick 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fe69806c7d0 msgr2=0x7fe69806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fe69806c7d0 0x7fe69806ec90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40b3460 tx=0x7fe6a0009190 comp rx=0 tx=0).stop 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 10 monclient: shutdown 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6b413fa50 msgr2=0x7fe6b41dafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6b413fa50 0x7fe6b41dafe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800c4a0 tx=0x7fe6a800c960 comp rx=0 tx=0).stop 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 shutdown_connections 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fe69806c7d0 0x7fe69806ec90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6b4146080 0x7fe6b41cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe6b4145840 0x7fe6b41cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 --2- 172.21.15.97:0/1918466796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6b413fa50 0x7fe6b41dafe0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 >> 172.21.15.97:0/1918466796 conn(0x7fe6b40b41f0 msgr2=0x7fe6b4144db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 shutdown_connections 2023-12-07T07:33:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.006+0000 7fe6bae10640 1 -- 172.21.15.97:0/1918466796 wait complete. 2023-12-07T07:33:42.012 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.007+0000 7fe6bae10640 1 librados: shutdown 2023-12-07T07:33:42.021 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-07T07:33:42.022 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-07T07:33:42.022 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-07T07:33:42.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-07T07:33:42.022 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-07T07:33:42.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.120+0000 7f73c3bb0640 10 monclient: get_monmap_and_config 2023-12-07T07:33:42.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.120+0000 7f73c3bb0640 10 monclient: build_initial_monmap 2023-12-07T07:33:42.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.120+0000 7f73c3bb0640 10 monclient: monmap: 2023-12-07T07:33:42.223 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:42.121892+0000 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:42.121892+0000 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:42.224 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.121+0000 7f73c3bb0640 1 Processor -- start 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.121+0000 7f73c3bb0640 1 -- start start 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.121+0000 7f73c3bb0640 10 monclient: init 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73bc0a8a30 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc142bf0 0x7f73bc13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient(hunting): picked mon.noname-b con 0x7f73bc142bf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73bc0a8e10 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc0aac10 con 0x7f73bc0a8a30 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc0aaef0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0aaa10 con 0x7f73bc142bf0 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c3bb0640 10 monclient(hunting): authenticate will time out at 1344.500618s 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c194c640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f73bc0a8a30 0x7f73bc1424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41182/0 (socket says 172.21.15.97:41182) 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.122+0000 7f73c194c640 1 -- 172.21.15.97:0/3356411477 learned_addr learned my addr 172.21.15.97:0/3356411477 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.123+0000 7f73c114b640 1 --2- 172.21.15.97:0/3356411477 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc142bf0 0x7f73bc13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.123+0000 7f73c114b640 10 monclient(hunting): get_auth_request con 0x7f73bc142bf0 auth_method 0 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.123+0000 7f73c114b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.123+0000 7f73c114b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.123+0000 7f73c114b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.124+0000 7f73c114b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.124+0000 7f73c114b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.124+0000 7f73c114b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.125+0000 7f73c094a640 1 -- 172.21.15.97:0/3356411477 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67644479 0 0) 0x7f73bc0aac10 con 0x7f73bc0a8a30 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.125+0000 7f73c094a640 10 monclient(hunting): _negotiate 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.125+0000 7f73c094a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.125+0000 7f73c094a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.125+0000 7f73c094a640 10 monclient(hunting): my global_id is 4254 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.125+0000 7f73c094a640 1 -- 172.21.15.97:0/3356411477 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a8003520 con 0x7f73bc0a8a30 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 1 -- 172.21.15.97:0/3356411477 >> v1:172.21.15.154:6789/0 conn(0x7f73bc0a8e10 legacy=0x7f73bc13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 1 -- 172.21.15.97:0/3356411477 >> v1:172.21.15.97:6789/0 conn(0x7f73bc0a8a30 legacy=0x7f73bc1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 10 monclient: _finish_hunting 0 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 1 monclient: found mon.noname-b 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.126+0000 7f73c114b640 1 -- 172.21.15.97:0/3356411477 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc0a97f0 con 0x7f73bc142bf0 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.129+0000 7f73c114b640 1 --2- 172.21.15.97:0/3356411477 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc142bf0 0x7f73bc13ecf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f73b80097d0 tx=0x7f73b802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e0d5bfb460ccb6 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:42.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 1 -- 172.21.15.97:0/3356411477 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8031890 con 0x7f73bc142bf0 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: dump: 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: _finish_auth 0 2023-12-07T07:33:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: _check_auth_tickets 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 1 -- 172.21.15.97:0/3356411477 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b80319f0 con 0x7f73bc142bf0 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 1 -- 172.21.15.97:0/3356411477 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8031d80 con 0x7f73bc142bf0 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c094a640 10 monclient: dump: 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:42.237 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c3bb0640 5 monclient: authenticate success, global_id 4241 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c3bb0640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.133+0000 7f73c3bb0640 4 set_mon_vals no callback set 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 10 monclient: shutdown 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3356411477 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc142bf0 msgr2=0x7f73bc13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3356411477 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc142bf0 0x7f73bc13ecf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f73b80097d0 tx=0x7f73b802b930 comp rx=0 tx=0).stop 2023-12-07T07:33:42.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3356411477 shutdown_connections 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3356411477 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc142bf0 0x7f73bc13ecf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3356411477 >> 172.21.15.97:0/3356411477 conn(0x7f73bc061a30 msgr2=0x7f73bc0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3356411477 shutdown_connections 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.134+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3356411477 wait complete. 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 10 monclient: build_initial_monmap 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 10 monclient: monmap: 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 1 librados: starting msgr at 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 1 librados: starting objecter 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 1 Processor -- start 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 1 -- start start 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 1 librados: setting wanted keys 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 1 librados: calling monclient init 2023-12-07T07:33:42.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.135+0000 7f73c3bb0640 10 monclient: init 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc0a8a30 0x7f73bc139d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): picked mon.noname-c con 0x7f73bc0a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0a8e10 0x7f73bc136ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): picked mon.noname-a con 0x7f73bc0a8e10 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc142bf0 0x7f73bc137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): picked mon.noname-b con 0x7f73bc142bf0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc13a8b0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc13b340 con 0x7f73bc142bf0 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc13b5b0 con 0x7f73bc0a8a30 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c3bb0640 10 monclient(hunting): authenticate will time out at 1344.514777s 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.136+0000 7f73c114b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0a8e10 0x7f73bc136ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc0a8a30 0x7f73bc139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0a8e10 0x7f73bc136ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38790/0 (socket says 172.21.15.97:38790) 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 1 -- 172.21.15.97:0/3641085916 learned_addr learned my addr 172.21.15.97:0/3641085916 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 10 monclient(hunting): get_auth_request con 0x7f73bc0a8e10 auth_method 0 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c214d640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc142bf0 0x7f73bc137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 10 monclient(hunting): get_auth_request con 0x7f73bc0a8a30 auth_method 0 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c214d640 10 monclient(hunting): get_auth_request con 0x7f73bc142bf0 auth_method 0 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c214d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c214d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c214d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c114b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.137+0000 7f73c194c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.138+0000 7f73c214d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.138+0000 7f73c214d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.138+0000 7f73c214d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 1 -- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc0a8a30 msgr2=0x7f73bc139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc0a8a30 0x7f73bc139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 1 -- 172.21.15.97:0/3641085916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc142bf0 msgr2=0x7f73bc137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc142bf0 0x7f73bc137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 10 monclient: _finish_hunting 0 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 1 monclient: found mon.noname-a 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.140+0000 7f73c114b640 1 -- 172.21.15.97:0/3641085916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b8009400 con 0x7f73bc0a8e10 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.143+0000 7f73c114b640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0a8e10 0x7f73bc136ac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73b80098f0 tx=0x7f73b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.143+0000 7f73b27fc640 10 client.?.objecter ms_handle_connect 0x7f73bc0a8e10 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.143+0000 7f73b27fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.146+0000 7f73b27fc640 1 -- 172.21.15.97:0/3641085916 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b803ad70 con 0x7f73bc0a8e10 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: dump: 2023-12-07T07:33:42.244 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 1 -- 172.21.15.97:0/3641085916 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b8045450 con 0x7f73bc0a8e10 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 5 monclient: authenticate success, global_id 4257 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 10 monclient: _renew_subs 2023-12-07T07:33:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73bc0b24e0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 1 -- 172.21.15.97:0/3641085916 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8045800 con 0x7f73bc0a8e10 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b27fc640 10 monclient: dump: 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73b3fff640 4 set_mon_vals no callback set 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 10 monclient: _renew_subs 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73bc1385f0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.147+0000 7f73c3bb0640 1 librados: init done 2023-12-07T07:33:42.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.148+0000 7f73c3bb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:42.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.148+0000 7f73c3bb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:42.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.148+0000 7f73c3bb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:42.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.148+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc067ac0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.440+0000 7f73b27fc640 1 -- 172.21.15.97:0/3641085916 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f73b80303e0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.441+0000 7f73b27fc640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f73a006c820 0x7f73a006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:42.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.441+0000 7f73c194c640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f73a006c820 0x7f73a006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:42.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.441+0000 7f73c194c640 10 monclient: get_auth_request con 0x7f73a006c820 auth_method 0 2023-12-07T07:33:42.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.441+0000 7f73b27fc640 10 client.4257.objecter ms_handle_connect 0x7f73a006c820 2023-12-07T07:33:42.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.441+0000 7f73c194c640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f73a006c820 0x7f73a006ece0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0011e0 tx=0x7f73ac007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 1 -- 172.21.15.97:0/3641085916 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f73b807ae80 con 0x7f73bc0a8e10 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 10 client.4257.objecter ms_dispatch 0x7f73bc000b90 osd_map(22..22 src has 1..22) v4 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 3 client.4257.objecter handle_osd_map got epochs [22,22] > 0 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 3 client.4257.objecter handle_osd_map decoding full epoch 22 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 20 client.4257.objecter dump_active .. 0 homeless 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 1 -- 172.21.15.97:0/3641085916 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73b80771c0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.455+0000 7f73b27fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:42.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.705+0000 7f73c3bb0640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-12-07T07:33:42.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.705+0000 7f73c3bb0640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-07T07:33:42.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.705+0000 7f73c3bb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:42.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.705+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f73bc141ff0 con 0x7f73bc0a8e10 2023-12-07T07:33:42.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.844+0000 7f73b27fc640 1 -- 172.21.15.97:0/3641085916 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171030 (secure 0 0 0) 0x7f73b8035020 con 0x7f73bc0a8e10 2023-12-07T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.844+0000 7f73b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-07T07:33:42.847 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.844+0000 7f73b27fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:42.849 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 10 librados: watch_flush enter 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 10 librados: watch_flush exit 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 20 client.4257.objecter shutdown clearing up homeless session... 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 10 client.4257.objecter successfully canceled tick 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f73a006c820 msgr2=0x7f73a006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f73a006c820 0x7f73a006ece0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0011e0 tx=0x7f73ac007d30 comp rx=0 tx=0).stop 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 10 monclient: shutdown 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0a8e10 msgr2=0x7f73bc136ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:42.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0a8e10 0x7f73bc136ac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73b80098f0 tx=0x7f73b80046c0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 shutdown_connections 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f73a006c820 0x7f73a006ece0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc142bf0 0x7f73bc137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f73bc0a8e10 0x7f73bc136ac0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 --2- 172.21.15.97:0/3641085916 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f73bc0a8a30 0x7f73bc139d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 >> 172.21.15.97:0/3641085916 conn(0x7f73bc061a30 msgr2=0x7f73bc0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 shutdown_connections 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 -- 172.21.15.97:0/3641085916 wait complete. 2023-12-07T07:33:42.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.847+0000 7f73c3bb0640 1 librados: shutdown 2023-12-07T07:33:42.861 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3287182838},{"type":"v1","addr":"172.21.15.97:6801","nonce":3287182838}]},"active_addr":"172.21.15.97:6801/3287182838","active_change":"2023-12-07T07:32:51.056190+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":205889932}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":2120486616}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":337811251}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":3342855449}]}]} 2023-12-07T07:33:42.865 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-07T07:33:42.865 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-07T07:33:42.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:42.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.967+0000 7f568ef4e640 10 monclient: get_monmap_and_config 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.967+0000 7f568ef4e640 10 monclient: build_initial_monmap 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.967+0000 7f568ef4e640 10 monclient: monmap: 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:42.968675+0000 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:42.968675+0000 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:42.970 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:42.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 1 Processor -- start 2023-12-07T07:33:42.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 1 -- start start 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 10 monclient: init 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56880a8910 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56880a9b70 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f568813f370 0x7f56881426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.968+0000 7f568ef4e640 10 monclient(hunting): picked mon.noname-c con 0x7f568813f370 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.969+0000 7f568ef4e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56880a8cf0 con 0x7f56880a9b70 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.969+0000 7f568ef4e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56880aabd0 con 0x7f56880a8910 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.969+0000 7f568ef4e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.969+0000 7f568ef4e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56880aaeb0 con 0x7f568813f370 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.969+0000 7f568ef4e640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:42.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:42.969+0000 7f568ef4e640 10 monclient(hunting): authenticate will time out at 1345.347335s 2023-12-07T07:33:43.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.008+0000 7f568d4eb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f568813f370 0x7f56881426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:43.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.010+0000 7f568ccea640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f56880a8910 0x7f568813ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44684/0 (socket says 172.21.15.97:44684) 2023-12-07T07:33:43.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.010+0000 7f568ccea640 1 -- 172.21.15.97:0/3724559101 learned_addr learned my addr 172.21.15.97:0/3724559101 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:43.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.010+0000 7f568d4eb640 10 monclient(hunting): get_auth_request con 0x7f568813f370 auth_method 0 2023-12-07T07:33:43.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.010+0000 7f568d4eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:43.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.010+0000 7f568d4eb640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:43.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.010+0000 7f568d4eb640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:43.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.011+0000 7f568d4eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:43.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.011+0000 7f568d4eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:43.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.011+0000 7f568d4eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:43.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-12-07T07:33:43.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 1 -- 172.21.15.97:0/3724559101 >> v1:172.21.15.154:6789/0 conn(0x7f56880a8910 legacy=0x7f568813ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:43.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 1 -- 172.21.15.97:0/3724559101 >> v1:172.21.15.97:6789/0 conn(0x7f56880a9b70 legacy=0x7f5688141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:43.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 10 monclient: _finish_hunting 0 2023-12-07T07:33:43.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 1 monclient: found mon.noname-c 2023-12-07T07:33:43.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:43.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:43.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.015+0000 7f568d4eb640 1 -- 172.21.15.97:0/3724559101 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56880a9720 con 0x7f568813f370 2023-12-07T07:33:43.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.021+0000 7f568d4eb640 1 --2- 172.21.15.97:0/3724559101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f568813f370 0x7f56881426a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5680009d90 tx=0x7f568002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15538a766a010148 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:43.027 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 1 -- 172.21.15.97:0/3724559101 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568002cd00 con 0x7f568813f370 2023-12-07T07:33:43.027 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:43.027 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:43.027 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: dump: 2023-12-07T07:33:43.027 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:43.027 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 1 -- 172.21.15.97:0/3724559101 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568002ce60 con 0x7f568813f370 2023-12-07T07:33:43.028 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 1 -- 172.21.15.97:0/3724559101 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680034660 con 0x7f568813f370 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f56877fe640 10 monclient: dump: 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.029 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f568ef4e640 5 monclient: authenticate success, global_id 4291 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f568ef4e640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:43.030 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.025+0000 7f568ef4e640 4 set_mon_vals no callback set 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 10 monclient: shutdown 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 1 -- 172.21.15.97:0/3724559101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f568813f370 msgr2=0x7f56881426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 1 --2- 172.21.15.97:0/3724559101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f568813f370 0x7f56881426a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5680009d90 tx=0x7f568002bf50 comp rx=0 tx=0).stop 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 1 -- 172.21.15.97:0/3724559101 shutdown_connections 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 1 --2- 172.21.15.97:0/3724559101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f568813f370 0x7f56881426a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 1 -- 172.21.15.97:0/3724559101 >> 172.21.15.97:0/3724559101 conn(0x7f56880b7780 msgr2=0x7f56880b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 1 -- 172.21.15.97:0/3724559101 shutdown_connections 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 1 -- 172.21.15.97:0/3724559101 wait complete. 2023-12-07T07:33:43.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 10 monclient: build_initial_monmap 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.026+0000 7f568ef4e640 10 monclient: monmap: 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:43.036 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:43.037 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:43.037 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:43.038 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:43.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 librados: starting msgr at 2023-12-07T07:33:43.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 librados: starting objecter 2023-12-07T07:33:43.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 Processor -- start 2023-12-07T07:33:43.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 -- start start 2023-12-07T07:33:43.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 librados: setting wanted keys 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 librados: calling monclient init 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 10 monclient: init 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a8910 0x7f5688139930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 10 monclient(hunting): picked mon.noname-b con 0x7f56880a8910 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f568813f370 0x7f56880b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 10 monclient(hunting): picked mon.noname-c con 0x7f568813f370 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.027+0000 7f568ef4e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5688137dd0 0x7f568813a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 10 monclient(hunting): picked mon.noname-a con 0x7f5688137dd0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5688139f30 con 0x7f5688137dd0 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568813bb20 con 0x7f56880a8910 2023-12-07T07:33:43.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56880b6b20 con 0x7f568813f370 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568ef4e640 10 monclient(hunting): authenticate will time out at 1345.406329s 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f5687fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f568813f370 0x7f56880b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f568d4eb640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5688137dd0 0x7f568813a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f5687fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f568813f370 0x7f56880b65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:37940/0 (socket says 172.21.15.97:37940) 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f5687fff640 1 -- 172.21.15.97:0/1123406724 learned_addr learned my addr 172.21.15.97:0/1123406724 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f5687fff640 10 monclient(hunting): get_auth_request con 0x7f568813f370 auth_method 0 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f5687fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f5687fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.028+0000 7f5687fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568d4eb640 10 monclient(hunting): get_auth_request con 0x7f5688137dd0 auth_method 0 2023-12-07T07:33:43.040 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568d4eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568d4eb640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568d4eb640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568ccea640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a8910 0x7f5688139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f5687fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f5687fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f5687fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568ccea640 10 monclient(hunting): get_auth_request con 0x7f56880a8910 auth_method 0 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568ccea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568ccea640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568ccea640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568d4eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568d4eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.029+0000 7f568d4eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.030+0000 7f568ccea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:43.041 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.030+0000 7f568ccea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.030+0000 7f568ccea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 1 -- 172.21.15.97:0/1123406724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a8910 msgr2=0x7f5688139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a8910 0x7f5688139930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 1 -- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5688137dd0 msgr2=0x7f568813a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5688137dd0 0x7f568813a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 1 monclient: found mon.noname-c 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:43.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.032+0000 7f5687fff640 1 -- 172.21.15.97:0/1123406724 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5680009a70 con 0x7f568813f370 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.034+0000 7f5687fff640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f568813f370 0x7f56880b65e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f567400c480 tx=0x7f567400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.034+0000 7f56857fa640 10 client.?.objecter ms_handle_connect 0x7f568813f370 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.034+0000 7f56857fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 1 -- 172.21.15.97:0/1123406724 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56740052d0 con 0x7f568813f370 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: dump: 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.044 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: _finish_auth 0 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 1 -- 172.21.15.97:0/1123406724 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5674005430 con 0x7f568813f370 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 5 monclient: authenticate success, global_id 4247 2023-12-07T07:33:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 10 monclient: _renew_subs 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 1 -- 172.21.15.97:0/1123406724 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5674012870 con 0x7f568813f370 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56880b7280 con 0x7f568813f370 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f56857fa640 10 monclient: dump: 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:43.046 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f5686ffd640 4 set_mon_vals no callback set 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 10 monclient: _renew_subs 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568813ab40 con 0x7f568813f370 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.038+0000 7f568ef4e640 1 librados: init done 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.039+0000 7f568ef4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.039+0000 7f568ef4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.039+0000 7f568ef4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:43.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.039+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568813ad60 con 0x7f568813f370 2023-12-07T07:33:43.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.057+0000 7f56857fa640 1 -- 172.21.15.97:0/1123406724 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f56740129d0 con 0x7f568813f370 2023-12-07T07:33:43.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.057+0000 7f56857fa640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f566c06c820 0x7f566c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:43.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.058+0000 7f568ccea640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f566c06c820 0x7f566c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:43.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.058+0000 7f568ccea640 10 monclient: get_auth_request con 0x7f566c06c820 auth_method 0 2023-12-07T07:33:43.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.058+0000 7f568ccea640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f566c06c820 0x7f566c06ece0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f567c005880 tx=0x7f567c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:43.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.058+0000 7f56857fa640 10 client.4247.objecter ms_handle_connect 0x7f566c06c820 2023-12-07T07:33:43.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.061+0000 7f56857fa640 1 -- 172.21.15.97:0/1123406724 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5674058920 con 0x7f568813f370 2023-12-07T07:33:43.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.061+0000 7f56857fa640 10 client.4247.objecter ms_dispatch 0x7f5688000b90 osd_map(23..23 src has 1..23) v4 2023-12-07T07:33:43.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.061+0000 7f56857fa640 3 client.4247.objecter handle_osd_map got epochs [23,23] > 0 2023-12-07T07:33:43.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.061+0000 7f56857fa640 3 client.4247.objecter handle_osd_map decoding full epoch 23 2023-12-07T07:33:43.064 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.061+0000 7f56857fa640 20 client.4247.objecter dump_active .. 0 homeless 2023-12-07T07:33:43.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.323+0000 7f56857fa640 1 -- 172.21.15.97:0/1123406724 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5674054c60 con 0x7f568813f370 2023-12-07T07:33:43.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.323+0000 7f56857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:43.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.323+0000 7f56857fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:43.472 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:09.922 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.517+0000 7f568ef4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:43.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.517+0000 7f568ef4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:43.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.517+0000 7f568ef4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:43.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:43.517+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5688075b40 con 0x7f568813f370 2023-12-07T07:33:43.526 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:09.976 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.539 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:09.977 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.560 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:10.009 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.596 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:10.034 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.620 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:10.070 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.783 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:10.232 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.845 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:10.283 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:43.869 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:10.319 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:33:44.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.104+0000 7f56857fa640 1 -- 172.21.15.97:0/1123406724 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14663 (secure 0 0 0) 0x7f5674058280 con 0x7f568813f370 2023-12-07T07:33:44.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.104+0000 7f56857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:44.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.104+0000 7f56857fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:44.107 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:33:44.107 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":23,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:33:42.933254+0000","last_up_change":"2023-12-07T07:33:33.953336+0000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T07:33:30.063715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T07:33:40.220751+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6819","nonce":2798215727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6821","nonce":2798215727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6825","nonce":2798215727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6823","nonce":2798215727}]},"public_addr":"172.21.15.97:6819/2798215727","cluster_addr":"172.21.15.97:6821/2798215727","heartbeat_back_addr":"172.21.15.97:6825/2798215727","heartbeat_front_addr":"172.21.15.97:6823/2798215727","state":["exists","up"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6827","nonce":138440183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6829","nonce":138440183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6833","nonce":138440183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6831","nonce":138440183}]},"public_addr":"172.21.15.97:6827/138440183","cluster_addr":"172.21.15.97:6829/138440183","heartbeat_back_addr":"172.21.15.97:6833/138440183","heartbeat_front_addr":"172.21.15.97:6831/138440183","state":["exists","up"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6803","nonce":1673674356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6805","nonce":1673674356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6809","nonce":1673674356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6807","nonce":1673674356}]},"public_addr":"172.21.15.97:6803/1673674356","cluster_addr":"172.21.15.97:6805/1673674356","heartbeat_back_addr":"172.21.15.97:6809/1673674356","heartbeat_front_addr":"172.21.15.97:6807/1673674356","state":["exists","up"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6811","nonce":929626392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6813","nonce":929626392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6817","nonce":929626392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6815","nonce":929626392}]},"public_addr":"172.21.15.97:6811/929626392","cluster_addr":"172.21.15.97:6813/929626392","heartbeat_back_addr":"172.21.15.97:6817/929626392","heartbeat_front_addr":"172.21.15.97:6815/929626392","state":["exists","up"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6809","nonce":1781054720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6811","nonce":1781054720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6815","nonce":1781054720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6813","nonce":1781054720}]},"public_addr":"172.21.15.154:6809/1781054720","cluster_addr":"172.21.15.154:6811/1781054720","heartbeat_back_addr":"172.21.15.154:6815/1781054720","heartbeat_front_addr":"172.21.15.154:6813/1781054720","state":["exists","up"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6801","nonce":4174110211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6803","nonce":4174110211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6807","nonce":4174110211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6805","nonce":4174110211}]},"public_addr":"172.21.15.154:6801/4174110211","cluster_addr":"172.21.15.154:6803/4174110211","heartbeat_back_addr":"172.21.15.154:6807/4174110211","heartbeat_front_addr":"172.21.15.154:6805/4174110211","state":["exists","up"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6825","nonce":1615374996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6827","nonce":1615374996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6831","nonce":1615374996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6829","nonce":1615374996}]},"public_addr":"172.21.15.154:6825/1615374996","cluster_addr":"172.21.15.154:6827/1615374996","heartbeat_back_addr":"172.21.15.154:6831/1615374996","heartbeat_front_addr":"172.21.15.154:6829/1615374996","state":["exists","up"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6817","nonce":1161771449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6819","nonce":1161771449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6823","nonce":1161771449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6821","nonce":1161771449}]},"public_addr":"172.21.15.154:6817/1161771449","cluster_addr":"172.21.15.154:6819/1161771449","heartbeat_back_addr":"172.21.15.154:6823/1161771449","heartbeat_front_addr":"172.21.15.154:6821/1161771449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.927364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.534718+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.623805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.804774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.252422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:28.225222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.956517+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:44.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 10 librados: watch_flush enter 2023-12-07T07:33:44.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 10 librados: watch_flush exit 2023-12-07T07:33:44.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 20 client.4247.objecter shutdown clearing up homeless session... 2023-12-07T07:33:44.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 10 client.4247.objecter successfully canceled tick 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f566c06c820 msgr2=0x7f566c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f566c06c820 0x7f566c06ece0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f567c005880 tx=0x7f567c009040 comp rx=0 tx=0).stop 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 10 monclient: shutdown 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f568813f370 msgr2=0x7f56880b65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f568813f370 0x7f56880b65e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f567400c480 tx=0x7f567400c940 comp rx=0 tx=0).stop 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 shutdown_connections 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f566c06c820 0x7f566c06ece0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5688137dd0 0x7f568813a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f568813f370 0x7f56880b65e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 --2- 172.21.15.97:0/1123406724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a8910 0x7f5688139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 >> 172.21.15.97:0/1123406724 conn(0x7f56880b7780 msgr2=0x7f568813c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:44.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.106+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 shutdown_connections 2023-12-07T07:33:44.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.107+0000 7f568ef4e640 1 -- 172.21.15.97:0/1123406724 wait complete. 2023-12-07T07:33:44.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.107+0000 7f568ef4e640 1 librados: shutdown 2023-12-07T07:33:44.122 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-07T07:33:44.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.224+0000 7f6594916640 10 monclient: get_monmap_and_config 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.224+0000 7f6594916640 10 monclient: build_initial_monmap 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.225+0000 7f6594916640 10 monclient: monmap: 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:44.226048+0000 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:44.226048+0000 2023-12-07T07:33:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:44.228 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:44.228 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:44.228 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:44.228 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:44.228 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:44.228 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:44.228 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:44.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.225+0000 7f6594916640 1 Processor -- start 2023-12-07T07:33:44.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.225+0000 7f6594916640 1 -- start start 2023-12-07T07:33:44.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.225+0000 7f6594916640 10 monclient: init 2023-12-07T07:33:44.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:44.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-12-07T07:33:44.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65900a8a30 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6590142bf0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f659013f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient(hunting): picked mon.noname-b con 0x7f65900a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65900aac10 con 0x7f6590142bf0 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65900aaef0 con 0x7f65900a8a30 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65900aaa10 con 0x7f65900a8e10 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.226+0000 7f6594916640 10 monclient(hunting): authenticate will time out at 1346.604737s 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.227+0000 7f658e59c640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f65900a8a30 0x7f65901424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44686/0 (socket says 172.21.15.97:44686) 2023-12-07T07:33:44.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.227+0000 7f658e59c640 1 -- 172.21.15.97:0/2606237111 learned_addr learned my addr 172.21.15.97:0/2606237111 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:44.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.275+0000 7f658ed9d640 1 --2- 172.21.15.97:0/2606237111 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f659013f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.275+0000 7f658ed9d640 10 monclient(hunting): get_auth_request con 0x7f65900a8e10 auth_method 0 2023-12-07T07:33:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.275+0000 7f658ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.275+0000 7f658ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.275+0000 7f658ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.276+0000 7f658ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.276+0000 7f658ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:44.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.276+0000 7f658ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 1 -- 172.21.15.97:0/2606237111 >> v1:172.21.15.154:6789/0 conn(0x7f65900a8a30 legacy=0x7f65901424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 1 -- 172.21.15.97:0/2606237111 >> v1:172.21.15.97:6789/0 conn(0x7f6590142bf0 legacy=0x7f659013ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 1 monclient: found mon.noname-b 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:44.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.278+0000 7f658ed9d640 1 -- 172.21.15.97:0/2606237111 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65900a97f0 con 0x7f65900a8e10 2023-12-07T07:33:44.282 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.280+0000 7f658ed9d640 1 --2- 172.21.15.97:0/2606237111 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f659013f400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6588009dc0 tx=0x7f658802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=841f3f9eae98d262 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 1 -- 172.21.15.97:0/2606237111 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658802ccd0 con 0x7f65900a8e10 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: dump: 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:44.286 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: _finish_auth 0 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: _check_auth_tickets 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 1 -- 172.21.15.97:0/2606237111 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658802ce30 con 0x7f65900a8e10 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f6594916640 5 monclient: authenticate success, global_id 4250 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f6594916640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f6594916640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 1 -- 172.21.15.97:0/2606237111 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588034660 con 0x7f65900a8e10 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f6594916640 4 set_mon_vals no callback set 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.283+0000 7f658dd9b640 10 monclient: dump: 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:44.287 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:44.288 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:44.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 10 monclient: shutdown 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 1 -- 172.21.15.97:0/2606237111 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 msgr2=0x7f659013f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 1 --2- 172.21.15.97:0/2606237111 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f659013f400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6588009dc0 tx=0x7f658802bf20 comp rx=0 tx=0).stop 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 1 -- 172.21.15.97:0/2606237111 shutdown_connections 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 1 --2- 172.21.15.97:0/2606237111 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f659013f400 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 1 -- 172.21.15.97:0/2606237111 >> 172.21.15.97:0/2606237111 conn(0x7f6590061a30 msgr2=0x7f65900b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 1 -- 172.21.15.97:0/2606237111 shutdown_connections 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.284+0000 7f6594916640 1 -- 172.21.15.97:0/2606237111 wait complete. 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.285+0000 7f6594916640 10 monclient: build_initial_monmap 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.285+0000 7f6594916640 10 monclient: monmap: 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:44.292 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 librados: starting msgr at 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 librados: starting objecter 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 Processor -- start 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 -- start start 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 librados: setting wanted keys 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 librados: calling monclient init 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient: init 2023-12-07T07:33:44.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65900a8a30 0x7f65900ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): picked mon.noname-b con 0x7f65900a8a30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f65900ab410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): picked mon.noname-c con 0x7f65900a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6590142bf0 0x7f65900ab950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): picked mon.noname-a con 0x7f6590142bf0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65900afee0 con 0x7f6590142bf0 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65900ad9b0 con 0x7f65900a8a30 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65900aec10 con 0x7f65900a8e10 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:44.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.286+0000 7f6594916640 10 monclient(hunting): authenticate will time out at 1346.665120s 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f6587fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f65900ab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f658e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65900a8a30 0x7f65900ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f6587fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f65900ab410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:37956/0 (socket says 172.21.15.97:37956) 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f6587fff640 1 -- 172.21.15.97:0/3617587232 learned_addr learned my addr 172.21.15.97:0/3617587232 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f6587fff640 10 monclient(hunting): get_auth_request con 0x7f65900a8e10 auth_method 0 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f6587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f6587fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f6587fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f658ed9d640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6590142bf0 0x7f65900ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:44.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f658e59c640 10 monclient(hunting): get_auth_request con 0x7f65900a8a30 auth_method 0 2023-12-07T07:33:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f658e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f658e59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.287+0000 7f658e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f6587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f6587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f6587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:44.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f658ed9d640 10 monclient(hunting): get_auth_request con 0x7f6590142bf0 auth_method 0 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f658ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f658ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f658ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f658e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f658e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.288+0000 7f658e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.289+0000 7f658ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.289+0000 7f658ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:44.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.289+0000 7f658ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:44.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.290+0000 7f658e59c640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2023-12-07T07:33:44.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.290+0000 7f658e59c640 1 -- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 msgr2=0x7f65900ab410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:44.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.290+0000 7f658e59c640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f65900ab410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:44.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.291+0000 7f658e59c640 1 -- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6590142bf0 msgr2=0x7f65900ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:44.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.291+0000 7f658e59c640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6590142bf0 0x7f65900ab950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:44.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.291+0000 7f658e59c640 10 monclient: _finish_hunting 0 2023-12-07T07:33:44.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.291+0000 7f658e59c640 1 monclient: found mon.noname-b 2023-12-07T07:33:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.291+0000 7f658e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.291+0000 7f658e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.291+0000 7f658e59c640 1 -- 172.21.15.97:0/3617587232 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588009a70 con 0x7f65900a8a30 2023-12-07T07:33:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.293+0000 7f658e59c640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65900a8a30 0x7f65900ae4b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f658000c440 tx=0x7f6580001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.293+0000 7f6586ffd640 10 client.?.objecter ms_handle_connect 0x7f65900a8a30 2023-12-07T07:33:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.293+0000 7f6586ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 1 -- 172.21.15.97:0/3617587232 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580005ce0 con 0x7f65900a8a30 2023-12-07T07:33:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: dump: 2023-12-07T07:33:44.300 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 1 -- 172.21.15.97:0/3617587232 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6580005e40 con 0x7f65900a8a30 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6594916640 5 monclient: authenticate success, global_id 4300 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6594916640 10 monclient: _renew_subs 2023-12-07T07:33:44.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6594916640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65900aeef0 con 0x7f65900a8a30 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 1 -- 172.21.15.97:0/3617587232 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800072d0 con 0x7f65900a8a30 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f6586ffd640 10 monclient: dump: 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:44.302 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.296+0000 7f658cd99640 4 set_mon_vals no callback set 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 10 monclient: _renew_subs 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65900ace60 con 0x7f65900a8a30 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 1 librados: init done 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:44.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.297+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65900ae9f0 con 0x7f65900a8a30 2023-12-07T07:33:44.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.315+0000 7f6586ffd640 1 -- 172.21.15.97:0/3617587232 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f65800074f0 con 0x7f65900a8a30 2023-12-07T07:33:44.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.315+0000 7f6586ffd640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f657006c820 0x7f657006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:44.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.315+0000 7f6587fff640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f657006c820 0x7f657006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:44.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.315+0000 7f6587fff640 10 monclient: get_auth_request con 0x7f657006c820 auth_method 0 2023-12-07T07:33:44.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.316+0000 7f6586ffd640 10 client.4300.objecter ms_handle_connect 0x7f657006c820 2023-12-07T07:33:44.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.316+0000 7f6587fff640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f657006c820 0x7f657006ece0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65900b7a30 tx=0x7f6578009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:44.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.318+0000 7f6586ffd640 1 -- 172.21.15.97:0/3617587232 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6580057e60 con 0x7f65900a8a30 2023-12-07T07:33:44.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.318+0000 7f6586ffd640 10 client.4300.objecter ms_dispatch 0x7f6590000b90 osd_map(23..23 src has 1..23) v4 2023-12-07T07:33:44.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.318+0000 7f6586ffd640 3 client.4300.objecter handle_osd_map got epochs [23,23] > 0 2023-12-07T07:33:44.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.318+0000 7f6586ffd640 3 client.4300.objecter handle_osd_map decoding full epoch 23 2023-12-07T07:33:44.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.318+0000 7f6586ffd640 20 client.4300.objecter dump_active .. 0 homeless 2023-12-07T07:33:44.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.558+0000 7f6586ffd640 1 -- 172.21.15.97:0/3617587232 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65800541a0 con 0x7f65900a8a30 2023-12-07T07:33:44.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.558+0000 7f6586ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:44.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.558+0000 7f6586ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.747+0000 7f6594916640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.747+0000 7f6594916640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.747+0000 7f6594916640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:44.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:44.747+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6590067ac0 con 0x7f65900a8a30 2023-12-07T07:33:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.124+0000 7f6586ffd640 1 -- 172.21.15.97:0/3617587232 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14650 (secure 0 0 0) 0x7f65800577c0 con 0x7f65900a8a30 2023-12-07T07:33:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.124+0000 7f6586ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:33:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.124+0000 7f6586ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:33:45.127 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":24,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:33:44.065971+0000","last_up_change":"2023-12-07T07:33:33.953336+0000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T07:33:30.063715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T07:33:40.220751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6819","nonce":2798215727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6821","nonce":2798215727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6825","nonce":2798215727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6823","nonce":2798215727}]},"public_addr":"172.21.15.97:6819/2798215727","cluster_addr":"172.21.15.97:6821/2798215727","heartbeat_back_addr":"172.21.15.97:6825/2798215727","heartbeat_front_addr":"172.21.15.97:6823/2798215727","state":["exists","up"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6827","nonce":138440183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6829","nonce":138440183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6833","nonce":138440183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6831","nonce":138440183}]},"public_addr":"172.21.15.97:6827/138440183","cluster_addr":"172.21.15.97:6829/138440183","heartbeat_back_addr":"172.21.15.97:6833/138440183","heartbeat_front_addr":"172.21.15.97:6831/138440183","state":["exists","up"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6803","nonce":1673674356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6805","nonce":1673674356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6809","nonce":1673674356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6807","nonce":1673674356}]},"public_addr":"172.21.15.97:6803/1673674356","cluster_addr":"172.21.15.97:6805/1673674356","heartbeat_back_addr":"172.21.15.97:6809/1673674356","heartbeat_front_addr":"172.21.15.97:6807/1673674356","state":["exists","up"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6811","nonce":929626392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6813","nonce":929626392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6817","nonce":929626392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6815","nonce":929626392}]},"public_addr":"172.21.15.97:6811/929626392","cluster_addr":"172.21.15.97:6813/929626392","heartbeat_back_addr":"172.21.15.97:6817/929626392","heartbeat_front_addr":"172.21.15.97:6815/929626392","state":["exists","up"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6809","nonce":1781054720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6811","nonce":1781054720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6815","nonce":1781054720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6813","nonce":1781054720}]},"public_addr":"172.21.15.154:6809/1781054720","cluster_addr":"172.21.15.154:6811/1781054720","heartbeat_back_addr":"172.21.15.154:6815/1781054720","heartbeat_front_addr":"172.21.15.154:6813/1781054720","state":["exists","up"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6801","nonce":4174110211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6803","nonce":4174110211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6807","nonce":4174110211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6805","nonce":4174110211}]},"public_addr":"172.21.15.154:6801/4174110211","cluster_addr":"172.21.15.154:6803/4174110211","heartbeat_back_addr":"172.21.15.154:6807/4174110211","heartbeat_front_addr":"172.21.15.154:6805/4174110211","state":["exists","up"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6825","nonce":1615374996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6827","nonce":1615374996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6831","nonce":1615374996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6829","nonce":1615374996}]},"public_addr":"172.21.15.154:6825/1615374996","cluster_addr":"172.21.15.154:6827/1615374996","heartbeat_back_addr":"172.21.15.154:6831/1615374996","heartbeat_front_addr":"172.21.15.154:6829/1615374996","state":["exists","up"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6817","nonce":1161771449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6819","nonce":1161771449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6823","nonce":1161771449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6821","nonce":1161771449}]},"public_addr":"172.21.15.154:6817/1161771449","cluster_addr":"172.21.15.154:6819/1161771449","heartbeat_back_addr":"172.21.15.154:6823/1161771449","heartbeat_front_addr":"172.21.15.154:6821/1161771449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.927364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.534718+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.623805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.804774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.252422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:28.225222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.956517+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:33:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 10 librados: watch_flush enter 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 10 librados: watch_flush exit 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 20 client.4300.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 10 client.4300.objecter successfully canceled tick 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f657006c820 msgr2=0x7f657006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f657006c820 0x7f657006ece0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65900b7a30 tx=0x7f6578009040 comp rx=0 tx=0).stop 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 10 monclient: shutdown 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65900a8a30 msgr2=0x7f65900ae4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65900a8a30 0x7f65900ae4b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f658000c440 tx=0x7f6580001e70 comp rx=0 tx=0).stop 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 shutdown_connections 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f657006c820 0x7f657006ece0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6590142bf0 0x7f65900ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f65900a8e10 0x7f65900ab410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 --2- 172.21.15.97:0/3617587232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65900a8a30 0x7f65900ae4b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 >> 172.21.15.97:0/3617587232 conn(0x7f6590061a30 msgr2=0x7f659013cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 shutdown_connections 2023-12-07T07:33:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 -- 172.21.15.97:0/3617587232 wait complete. 2023-12-07T07:33:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.127+0000 7f6594916640 1 librados: shutdown 2023-12-07T07:33:45.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-07T07:33:45.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-07T07:33:45.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-07T07:33:45.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-07T07:33:45.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-07T07:33:45.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-07T07:33:45.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-07T07:33:45.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-07T07:33:45.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.329+0000 7fa11ffff640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.329+0000 7fa11ffff640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.329+0000 7fa11ffff640 10 monclient: monmap: 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.330989+0000 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.330989+0000 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.333 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.334 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.334 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.330+0000 7fa11ffff640 1 Processor -- start 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 1 -- start start 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 10 monclient: init 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa118038fe0 0x7fa1180c9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fa118038fe0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 0x7fa1180cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.331+0000 7fa11ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fa118035640 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.342 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7f185256e640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.342 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7f185256e640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.342 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7f185256e640 10 monclient: monmap: 2023-12-07T07:33:45.342 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.342 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.342 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.340895+0000 2023-12-07T07:33:45.342 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.340895+0000 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.343 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.336+0000 7fa11ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1180cc600 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7fa11ffff640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa118035bf0 con 0x7fa1180cc600 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7fa11ffff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7fa11ffff640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1180393c0 con 0x7fa118038fe0 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7fa11ffff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7fa11ffff640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1180373e0 con 0x7fa118035640 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7fa11ffff640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.339+0000 7fa11ffff640 10 monclient(hunting): authenticate will time out at 1347.717364s 2023-12-07T07:33:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.338+0000 7f8447056640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.345 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11effd640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa118038fe0 0x7fa1180c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.345 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11effd640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa118038fe0 0x7fa1180c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38820/0 (socket says 172.21.15.97:38820) 2023-12-07T07:33:45.345 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11effd640 1 -- 172.21.15.97:0/2484661156 learned_addr learned my addr 172.21.15.97:0/2484661156 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.345 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11effd640 10 monclient(hunting): get_auth_request con 0x7fa118038fe0 auth_method 0 2023-12-07T07:33:45.345 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11effd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11e7fc640 1 --2- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 0x7fa1180cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11e7fc640 10 monclient(hunting): get_auth_request con 0x7fa118035640 auth_method 0 2023-12-07T07:33:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7fa11e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7f185256e640 1 Processor -- start 2023-12-07T07:33:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7f185256e640 1 -- start start 2023-12-07T07:33:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.341+0000 7f185256e640 10 monclient: init 2023-12-07T07:33:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-12-07T07:33:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f184c0b3c60 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f184c0b1750 0x7f184c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient(hunting): picked mon.noname-c con 0x7f184c0b1750 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c142e70 0x7f184c1432c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient(hunting): picked mon.noname-b con 0x7f184c142e70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c06f3b0 con 0x7f184c0b3c60 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c0b4040 con 0x7f184c0b1750 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.338+0000 7f8447056640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.338+0000 7f8447056640 10 monclient: monmap: 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.339819+0000 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.339819+0000 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.350 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.351 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c070b90 con 0x7f184c142e70 2023-12-07T07:33:45.351 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.351 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7f185256e640 10 monclient(hunting): authenticate will time out at 1347.720828s 2023-12-07T07:33:45.351 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.343+0000 7f8447056640 1 Processor -- start 2023-12-07T07:33:45.351 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.343+0000 7f184bfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f184c0b3c60 0x7f184c06e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44694/0 (socket says 172.21.15.97:44694) 2023-12-07T07:33:45.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.343+0000 7f184bfff640 1 -- 172.21.15.97:0/603311123 learned_addr learned my addr 172.21.15.97:0/603311123 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.343+0000 7f8447056640 1 -- start start 2023-12-07T07:33:45.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7fa11effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7fa11effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.342+0000 7fa11effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.343+0000 7f8447056640 10 monclient: init 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84400af360 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440143920 0x7f8440149880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient(hunting): picked mon.noname-b con 0x7f8440143920 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8440149dc0 0x7f84401509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient(hunting): picked mon.noname-c con 0x7f8440149dc0 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8440143d40 con 0x7f84400af360 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84400af740 con 0x7f8440149dc0 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8440070c50 con 0x7f8440143920 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7f8447056640 10 monclient(hunting): authenticate will time out at 1347.722411s 2023-12-07T07:33:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7fd4f0fb0640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7fd4f0fb0640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.344+0000 7fd4f0fb0640 10 monclient: monmap: 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.345620+0000 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.345620+0000 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.356 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.357 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7f1850b0b640 1 --2- 172.21.15.97:0/603311123 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c142e70 0x7f184c1432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7f1850b0b640 10 monclient(hunting): get_auth_request con 0x7f184c142e70 auth_method 0 2023-12-07T07:33:45.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7f1850b0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7f1850b0b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7f1850b0b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7fd4f0fb0640 1 Processor -- start 2023-12-07T07:33:45.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7fd4f0fb0640 1 -- start start 2023-12-07T07:33:45.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.345+0000 7fd4f0fb0640 10 monclient: init 2023-12-07T07:33:45.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7f184b7fe640 1 --2- 172.21.15.97:0/603311123 >> v2:172.21.15.154:3300/0 conn(0x7f184c0b1750 0x7f184c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7f184b7fe640 10 monclient(hunting): get_auth_request con 0x7f184c0b1750 auth_method 0 2023-12-07T07:33:45.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7f184b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7f184b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7f184b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.360 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4ec0af360 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4ec06f2b0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec14a360 0x7fd4ec151a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ec14a360 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec143e60 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec070fc0 con 0x7fd4ec0af360 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0af740 con 0x7fd4ec14a360 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.361 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.346+0000 7fd4f0fb0640 10 monclient(hunting): authenticate will time out at 1347.725223s 2023-12-07T07:33:45.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7f1850b0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7f1850b0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7f1850b0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7fa11e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7fa11e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.347+0000 7fa11e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 1 -- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 msgr2=0x7fa1180cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 1 --2- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 0x7fa1180cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 1 -- 172.21.15.97:0/2484661156 >> v1:172.21.15.154:6789/0 conn(0x7fa1180cc600 legacy=0x7fa118034420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 1 monclient: found mon.noname-a 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7fa11effd640 1 -- 172.21.15.97:0/2484661156 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa118037650 con 0x7fa118038fe0 2023-12-07T07:33:45.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7f8444df2640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f84400af360 0x7f8440143210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41196/0 (socket says 172.21.15.97:41196) 2023-12-07T07:33:45.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.348+0000 7f8444df2640 1 -- 172.21.15.97:0/318108486 learned_addr learned my addr 172.21.15.97:0/318108486 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7fd4e9d9b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd4ec06f2b0 0x7fd4ec149c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41212/0 (socket says 172.21.15.97:41212) 2023-12-07T07:33:45.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7fd4e9d9b640 1 -- 172.21.15.97:0/4153908151 learned_addr learned my addr 172.21.15.97:0/4153908151 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f84455f3640 1 --2- 172.21.15.97:0/318108486 >> v2:172.21.15.154:3300/0 conn(0x7f8440149dc0 0x7f84401509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184affd640 1 -- 172.21.15.97:0/603311123 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373429273 0 0) 0x7f184c06f3b0 con 0x7f184c0b3c60 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184affd640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184affd640 10 monclient(hunting): my global_id is 4306 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f184affd640 1 -- 172.21.15.97:0/603311123 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834003520 con 0x7f184c0b3c60 2023-12-07T07:33:45.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f84455f3640 10 monclient(hunting): get_auth_request con 0x7f8440149dc0 auth_method 0 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f84455f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f84455f3640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.349+0000 7f84455f3640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 1 -- 172.21.15.97:0/603311123 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c142e70 msgr2=0x7f184c1432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 1 --2- 172.21.15.97:0/603311123 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c142e70 0x7f184c1432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 1 -- 172.21.15.97:0/603311123 >> v1:172.21.15.154:6789/0 conn(0x7f184c0b3c60 legacy=0x7f184c06e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.352+0000 7f184b7fe640 1 -- 172.21.15.97:0/603311123 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c06f5e0 con 0x7f184c0b1750 2023-12-07T07:33:45.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.353+0000 7f843ffff640 1 --2- 172.21.15.97:0/318108486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440143920 0x7f8440149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.353+0000 7f84455f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.353+0000 7f84455f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.353+0000 7f84455f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.354+0000 7f532a659640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.354+0000 7f532a659640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.354+0000 7f532a659640 10 monclient: monmap: 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.368 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.355580+0000 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.355580+0000 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.369 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.355+0000 7fa11effd640 1 --2- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa118038fe0 0x7fa1180c9640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa114009870 tx=0x7fa11402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1192ec0746ac64 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.370 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.355+0000 7f532a659640 1 Processor -- start 2023-12-07T07:33:45.370 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.355+0000 7ffafb393640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.370 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.355+0000 7ffafb393640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.370 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.355+0000 7ffafb393640 10 monclient: monmap: 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.356784+0000 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.356784+0000 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.355+0000 7f843f7fe640 1 -- 172.21.15.97:0/318108486 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651059093 0 0) 0x7f8438001490 con 0x7f84400af360 2023-12-07T07:33:45.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843f7fe640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843f7fe640 10 monclient(hunting): my global_id is 4275 2023-12-07T07:33:45.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843f7fe640 1 -- 172.21.15.97:0/318108486 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8428003520 con 0x7f84400af360 2023-12-07T07:33:45.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f532a659640 1 -- start start 2023-12-07T07:33:45.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f532a659640 10 monclient: init 2023-12-07T07:33:45.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843ffff640 10 monclient(hunting): get_auth_request con 0x7f8440143920 auth_method 0 2023-12-07T07:33:45.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843ffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f843ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7ffafb393640 1 Processor -- start 2023-12-07T07:33:45.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7ffafb393640 1 -- start start 2023-12-07T07:33:45.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7ffafb393640 10 monclient: init 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.356+0000 7f532a659640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f531c006de0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f531c0071c0 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c033180 0x7f531c0d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 10 monclient(hunting): picked mon.noname-b con 0x7f531c033180 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c032590 con 0x7f531c006de0 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c0c3550 con 0x7f531c0071c0 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c032170 con 0x7f531c033180 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7f532a659640 10 monclient(hunting): authenticate will time out at 1347.735397s 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7fd4e959a640 1 -- 172.21.15.97:0/4153908151 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140926169 0 0) 0x7fd4ec070fc0 con 0x7fd4ec0af360 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7fd4e959a640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7fd4e959a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7fd4e959a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7fd4e959a640 10 monclient(hunting): my global_id is 4318 2023-12-07T07:33:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7fd4e959a640 1 -- 172.21.15.97:0/4153908151 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d4003520 con 0x7fd4ec0af360 2023-12-07T07:33:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7f184b7fe640 1 --2- 172.21.15.97:0/603311123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c0b1750 0x7f184c06eb90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f183c0097a0 tx=0x7f183c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=141da236fa935f2f server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7ffafb393640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7ffafb393640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-12-07T07:33:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7ffafb393640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffaec036df0 0x7ffaec0c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7ffafb393640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec036df0 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7ffafb393640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec0c3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7ffafb393640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec0355b0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.357+0000 7ffafb393640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ffaec0c8f00 0x7ffaec0c9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec0c8f00 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec0371d0 con 0x7ffaec0c8f00 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec02e6b0 con 0x7ffaec036df0 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec02e990 con 0x7ffaec0355b0 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.358+0000 7ffafb393640 10 monclient(hunting): authenticate will time out at 1347.736329s 2023-12-07T07:33:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.359+0000 7f5493b6c640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.359+0000 7f5493b6c640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.359+0000 7f5493b6c640 10 monclient: monmap: 2023-12-07T07:33:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.360634+0000 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.360634+0000 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f5493b6c640 1 Processor -- start 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f5493b6c640 1 -- start start 2023-12-07T07:33:45.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f5493b6c640 10 monclient: init 2023-12-07T07:33:45.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-12-07T07:33:45.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 1 -- 172.21.15.97:0/318108486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440143920 msgr2=0x7f8440149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 1 --2- 172.21.15.97:0/318108486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440143920 0x7f8440149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 1 -- 172.21.15.97:0/318108486 >> v1:172.21.15.97:6789/0 conn(0x7f84400af360 legacy=0x7f8440143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.360+0000 7f84455f3640 1 -- 172.21.15.97:0/318108486 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440070f30 con 0x7f8440149dc0 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5484038d70 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5484039150 0x7f54840c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient(hunting): picked mon.noname-b con 0x7f5484039150 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5484036480 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54840c7e60 con 0x7f5484036480 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54840c5930 con 0x7f5484038d70 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.383 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5484037600 con 0x7f5484039150 2023-12-07T07:33:45.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.361+0000 7f5493b6c640 10 monclient(hunting): authenticate will time out at 1347.739889s 2023-12-07T07:33:45.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.362+0000 7fd4ead9d640 1 --2- 172.21.15.97:0/4153908151 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec14a360 0x7fd4ec151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.363+0000 7fd4ead9d640 10 monclient(hunting): get_auth_request con 0x7fd4ec14a360 auth_method 0 2023-12-07T07:33:45.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.363+0000 7fd4ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.363+0000 7fd4ead9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.384 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.363+0000 7fd4ead9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.363+0000 7f84455f3640 1 --2- 172.21.15.97:0/318108486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440149dc0 0x7f84401509f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f84340097a0 tx=0x7f843402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3656d47a23fe1f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.364+0000 7f5491908640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f5484038d70 0x7f54840c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44712/0 (socket says 172.21.15.97:44712) 2023-12-07T07:33:45.385 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.364+0000 7f5491908640 1 -- 172.21.15.97:0/2565904305 learned_addr learned my addr 172.21.15.97:0/2565904305 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.364+0000 7fd4ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.364+0000 7fd4ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.364+0000 7fd4ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.364+0000 7f5328e56640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f531c0071c0 0x7f531c032a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44708/0 (socket says 172.21.15.97:44708) 2023-12-07T07:33:45.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.364+0000 7f5328e56640 1 -- 172.21.15.97:0/425832867 learned_addr learned my addr 172.21.15.97:0/425832867 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.386 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.365+0000 7f5329e58640 1 --2- 172.21.15.97:0/425832867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c033180 0x7f531c0d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.365+0000 7fd4e959a640 1 -- 172.21.15.97:0/4153908151 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815021478 0 0) 0x7fd4ec143e60 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.365+0000 7fd4e959a640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.365+0000 7fd4e959a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.365+0000 7fd4e959a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.365+0000 7fd4e959a640 10 monclient(hunting): my global_id is 4278 2023-12-07T07:33:45.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.365+0000 7fd4e959a640 1 -- 172.21.15.97:0/4153908151 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ec070fc0 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.387 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.366+0000 7fdb48ad0640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.366+0000 7fdb48ad0640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.366+0000 7fdb48ad0640 10 monclient: monmap: 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.367352+0000 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.367352+0000 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.388 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.366+0000 7f5329e58640 10 monclient(hunting): get_auth_request con 0x7f531c033180 auth_method 0 2023-12-07T07:33:45.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.366+0000 7f5329e58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.366+0000 7f5329e58640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.389 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.366+0000 7f5329e58640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 1 Processor -- start 2023-12-07T07:33:45.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 1 -- start start 2023-12-07T07:33:45.390 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 10 monclient: init 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb440b1490 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb4413cf10 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413fd60 0x7fdb441401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.367+0000 7fdb48ad0640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4413fd60 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fdb48ad0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb44072010 con 0x7fdb4413cf10 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fdb48ad0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb440724a0 con 0x7fdb440b1490 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fdb48ad0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fdb48ad0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb44071d80 con 0x7fdb4413fd60 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fdb48ad0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.391 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fdb48ad0640 10 monclient(hunting): authenticate will time out at 1347.746357s 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 1 -- 172.21.15.97:0/4153908151 >> v1:172.21.15.154:6789/0 conn(0x7fd4ec0af360 legacy=0x7fd4ec1469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 1 -- 172.21.15.97:0/4153908151 >> v1:172.21.15.97:6789/0 conn(0x7fd4ec06f2b0 legacy=0x7fd4ec149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.392 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7fd4ead9d640 1 -- 172.21.15.97:0/4153908151 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec149730 con 0x7fd4ec14a360 2023-12-07T07:33:45.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7f5329e58640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7f5329e58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.368+0000 7f5329e58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2023-12-07T07:33:45.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 1 -- 172.21.15.97:0/425832867 >> v1:172.21.15.154:6789/0 conn(0x7f531c0071c0 legacy=0x7f531c032a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 1 -- 172.21.15.97:0/425832867 >> v1:172.21.15.97:6789/0 conn(0x7f531c006de0 legacy=0x7f531c0c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.370+0000 7f5329e58640 1 -- 172.21.15.97:0/425832867 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c030d80 con 0x7f531c033180 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.372+0000 7ffafa391640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffaec036df0 0x7ffaec0c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.373+0000 7ffafa391640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffaec036df0 0x7ffaec0c2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38482/0 (socket says 172.21.15.97:38482) 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.373+0000 7ffafa391640 1 -- 172.21.15.97:0/2126005997 learned_addr learned my addr 172.21.15.97:0/2126005997 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.394 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.373+0000 7ffafa391640 10 monclient(hunting): get_auth_request con 0x7ffaec036df0 auth_method 0 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.373+0000 7ffafa391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.373+0000 7ffafa391640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.373+0000 7ffafa391640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.374+0000 7ffafa391640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.374+0000 7ffafa391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.374+0000 7ffafa391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.374+0000 7fd4ead9d640 1 --2- 172.21.15.97:0/4153908151 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec14a360 0x7fd4ec151a70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00097d0 tx=0x7fd4e002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c26f30e71e8bd server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.395 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.374+0000 7ffaf9b90640 1 --2- 172.21.15.97:0/2126005997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec0c3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.375+0000 7f5329e58640 1 --2- 172.21.15.97:0/425832867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c033180 0x7f531c0d1270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f53140097d0 tx=0x7f531402b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=345e2617d0b603f4 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.375+0000 7ffaf9b90640 10 monclient(hunting): get_auth_request con 0x7ffaec0355b0 auth_method 0 2023-12-07T07:33:45.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.375+0000 7ffaf9b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.376+0000 7ffaf9b90640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.376+0000 7ffaf9b90640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.376+0000 7fdb41d9b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdb4413cf10 0x7fdb4413d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41232/0 (socket says 172.21.15.97:41232) 2023-12-07T07:33:45.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.376+0000 7fdb41d9b640 1 -- 172.21.15.97:0/1518953953 learned_addr learned my addr 172.21.15.97:0/1518953953 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.376+0000 7fdb42d9d640 1 --2- 172.21.15.97:0/1518953953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413fd60 0x7fdb441401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7ffaf9b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7ffaf9b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7ffaf9b90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 1 -- 172.21.15.97:0/603311123 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c02e070 con 0x7f184c0b1750 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: dump: 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.398 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 1 -- 172.21.15.97:0/603311123 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c0315a0 con 0x7f184c0b1750 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f185256e640 5 monclient: authenticate success, global_id 4312 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f185256e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f185256e640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f185256e640 4 set_mon_vals no callback set 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 1 -- 172.21.15.97:0/603311123 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c031970 con 0x7f184c0b1750 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f184affd640 10 monclient: dump: 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.400 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7fdb42d9d640 10 monclient(hunting): get_auth_request con 0x7fdb4413fd60 auth_method 0 2023-12-07T07:33:45.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7fdb42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7fdb42d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.401 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7fdb42d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504680742 0 0) 0x7f54840c5930 con 0x7f5484038d70 2023-12-07T07:33:45.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f5490906640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.377+0000 7f5490906640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f5490906640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f5490906640 10 monclient(hunting): my global_id is 4321 2023-12-07T07:33:45.402 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5478003520 con 0x7f5484038d70 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 10 monclient: shutdown 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 -- 172.21.15.97:0/603311123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c0b1750 msgr2=0x7f184c06eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 --2- 172.21.15.97:0/603311123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c0b1750 0x7f184c06eb90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f183c0097a0 tx=0x7f183c02b840 comp rx=0 tx=0).stop 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 -- 172.21.15.97:0/603311123 shutdown_connections 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 --2- 172.21.15.97:0/603311123 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c142e70 0x7f184c1432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 --2- 172.21.15.97:0/603311123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c0b1750 0x7f184c06eb90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 -- 172.21.15.97:0/603311123 >> 172.21.15.97:0/603311123 conn(0x7f184c0b4a00 msgr2=0x7f184c0b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 -- 172.21.15.97:0/603311123 shutdown_connections 2023-12-07T07:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.378+0000 7f185256e640 1 -- 172.21.15.97:0/603311123 wait complete. 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 1 -- 172.21.15.97:0/2126005997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 msgr2=0x7ffaec0c3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 1 --2- 172.21.15.97:0/2126005997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec0c3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 1 -- 172.21.15.97:0/2126005997 >> v2:172.21.15.97:3300/0 conn(0x7ffaec0c8f00 msgr2=0x7ffaec0c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 1 --2- 172.21.15.97:0/2126005997 >> v2:172.21.15.97:3300/0 conn(0x7ffaec0c8f00 0x7ffaec0c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.404 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7ffafa391640 1 -- 172.21.15.97:0/2126005997 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec02cfb0 con 0x7ffaec036df0 2023-12-07T07:33:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 1 -- 172.21.15.97:0/2484661156 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114031640 con 0x7fa118038fe0 2023-12-07T07:33:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: dump: 2023-12-07T07:33:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.405 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 1 -- 172.21.15.97:0/2484661156 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1140317a0 con 0x7fa118038fe0 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.406 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 1 -- 172.21.15.97:0/2484661156 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114031b30 con 0x7fa118038fe0 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11dffb640 10 monclient: dump: 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.407 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7f185256e640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7f185256e640 10 monclient: monmap: 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.408 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11ffff640 5 monclient: authenticate success, global_id 4272 2023-12-07T07:33:45.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11ffff640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.409 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7fa11ffff640 4 set_mon_vals no callback set 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7f185256e640 1 librados: starting msgr at 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7f185256e640 1 librados: starting objecter 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.379+0000 7f185256e640 1 Processor -- start 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 -- start start 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 librados: setting wanted keys 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 librados: calling monclient init 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 10 monclient: shutdown 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.410 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 -- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa118038fe0 msgr2=0x7fa1180c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 --2- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa118038fe0 0x7fa1180c9640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa114009870 tx=0x7fa11402b8f0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient: init 2023-12-07T07:33:45.411 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 -- 172.21.15.97:0/2484661156 shutdown_connections 2023-12-07T07:33:45.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 --2- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 0x7fa1180cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 --2- 172.21.15.97:0/2484661156 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa118038fe0 0x7fa1180c9640 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 -- 172.21.15.97:0/2484661156 >> 172.21.15.97:0/2484661156 conn(0x7fa1180065d0 msgr2=0x7fa1180069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 -- 172.21.15.97:0/2484661156 shutdown_connections 2023-12-07T07:33:45.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7fa11ffff640 1 -- 172.21.15.97:0/2484661156 wait complete. 2023-12-07T07:33:45.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.412 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f184c0b1750 0x7f184c0a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): picked mon.noname-a con 0x7f184c0b1750 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c0b3c60 0x7f184c1dbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): picked mon.noname-c con 0x7f184c0b3c60 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c142e70 0x7f184c1dc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): picked mon.noname-b con 0x7f184c142e70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c1dc7d0 con 0x7f184c0b1750 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c1db830 con 0x7f184c142e70 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c14a510 con 0x7f184c0b3c60 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.380+0000 7f185256e640 10 monclient(hunting): authenticate will time out at 1347.759146s 2023-12-07T07:33:45.414 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 10 monclient: monmap: 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 1 librados: starting msgr at 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 1 librados: starting objecter 2023-12-07T07:33:45.415 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 1 Processor -- start 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 1 -- start start 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 1 librados: setting wanted keys 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 1 librados: calling monclient init 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.381+0000 7fa11ffff640 10 monclient: init 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 0x7fa118165030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fa118035640 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa118038fe0 0x7fa11802fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fa118038fe0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1180cc600 0x7fa11802d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fa1180cc600 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11802e990 con 0x7fa1180cc600 2023-12-07T07:33:45.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa118033600 con 0x7fa118038fe0 2023-12-07T07:33:45.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa118165c10 con 0x7fa118035640 2023-12-07T07:33:45.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7fa11ffff640 10 monclient(hunting): authenticate will time out at 1347.760648s 2023-12-07T07:33:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 1 -- 172.21.15.97:0/318108486 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843402d040 con 0x7f8440149dc0 2023-12-07T07:33:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 10 monclient: dump: 2023-12-07T07:33:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.418 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f8447056640 5 monclient: authenticate success, global_id 4315 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f8447056640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 1 -- 172.21.15.97:0/318108486 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8434031810 con 0x7f8440149dc0 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f843f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.419 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f8447056640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.382+0000 7f8447056640 4 set_mon_vals no callback set 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f843f7fe640 1 -- 172.21.15.97:0/318108486 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434031ba0 con 0x7f8440149dc0 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f843f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f843f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f843f7fe640 10 monclient: dump: 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.420 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7fdb42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7fdb42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7fdb42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 10 monclient: shutdown 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 -- 172.21.15.97:0/318108486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440149dc0 msgr2=0x7f84401509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 --2- 172.21.15.97:0/318108486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440149dc0 0x7f84401509f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f84340097a0 tx=0x7f843402b8b0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 -- 172.21.15.97:0/318108486 shutdown_connections 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 --2- 172.21.15.97:0/318108486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440149dc0 0x7f84401509f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 --2- 172.21.15.97:0/318108486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440143920 0x7f8440149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 -- 172.21.15.97:0/318108486 >> 172.21.15.97:0/318108486 conn(0x7f84400b01e0 msgr2=0x7f84400b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 -- 172.21.15.97:0/318108486 shutdown_connections 2023-12-07T07:33:45.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.383+0000 7f8447056640 1 -- 172.21.15.97:0/318108486 wait complete. 2023-12-07T07:33:45.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7fdb4159a640 1 -- 172.21.15.97:0/1518953953 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289945846 0 0) 0x7fdb44072010 con 0x7fdb4413cf10 2023-12-07T07:33:45.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7fdb4159a640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7fdb4159a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7fdb4159a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7fdb4159a640 10 monclient(hunting): my global_id is 4281 2023-12-07T07:33:45.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7fdb4159a640 1 -- 172.21.15.97:0/1518953953 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2c003520 con 0x7fdb4413cf10 2023-12-07T07:33:45.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7f5491107640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5484039150 0x7f54840c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7f8447056640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7f8447056640 10 monclient: monmap: 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.425 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7f8447056640 1 librados: starting msgr at 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.384+0000 7f8447056640 1 librados: starting objecter 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 Processor -- start 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 -- start start 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 librados: setting wanted keys 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 librados: calling monclient init 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient: init 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400af360 0x7f8440148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): picked mon.noname-b con 0x7f84400af360 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8440143920 0x7f8440148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): picked mon.noname-a con 0x7f8440143920 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440149dc0 0x7f84401db760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): picked mon.noname-c con 0x7f8440149dc0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400a92a0 con 0x7f8440143920 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400b1a40 con 0x7f84400af360 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84401472d0 con 0x7f8440149dc0 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.385+0000 7f8447056640 10 monclient(hunting): authenticate will time out at 1347.764282s 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.386+0000 7ffafa391640 1 --2- 172.21.15.97:0/2126005997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 0x7ffaec0c2bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4050140 tx=0x7ffaf40a5d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a62b335223d9147 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.386+0000 7fdb42d9d640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.386+0000 7fdb42d9d640 1 -- 172.21.15.97:0/1518953953 >> v1:172.21.15.154:6789/0 conn(0x7fdb440b1490 legacy=0x7fdb44071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.386+0000 7fdb42d9d640 1 -- 172.21.15.97:0/1518953953 >> v1:172.21.15.97:6789/0 conn(0x7fdb4413cf10 legacy=0x7fdb4413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.387+0000 7fdb42d9d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.387+0000 7fdb42d9d640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.387+0000 7fdb42d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.387+0000 7fdb42d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.387+0000 7fdb42d9d640 1 -- 172.21.15.97:0/1518953953 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb440b2040 con 0x7fdb4413fd60 2023-12-07T07:33:45.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.388+0000 7f843ffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8440143920 0x7f8440148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3366013203 0 0) 0x7f5478003520 con 0x7f5484038d70 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5484039150 msgr2=0x7f54840c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5484039150 0x7f54840c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 >> v1:172.21.15.97:6789/0 conn(0x7f5484036480 legacy=0x7f5484034420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 monclient: found mon.noname-c-legacy 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484035860 con 0x7f5484038d70 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2176119509 0 0) 0x7f5488001670 con 0x7f5484038d70 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5493b6c640 5 monclient: authenticate success, global_id 4321 2023-12-07T07:33:45.430 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: dump: 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2023-12-07T07:33:45.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 >> v1:172.21.15.154:6789/0 conn(0x7f5484038d70 legacy=0x7f54840c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:33:45.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5478003970 0x7f5478003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f184b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c0b3c60 0x7f184c1dbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): picked mon.a con 0x7f5478003970 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54780047c0 0x7f5478004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): picked mon.b con 0x7f54780047c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f547800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): picked mon.c con 0x7f5478005380 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5484037600 con 0x7f5478003970 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547800bff0 con 0x7f54780047c0 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547800c1b0 con 0x7f5478005380 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5490906640 10 monclient(hunting): _finish_auth 0 2023-12-07T07:33:45.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.389+0000 7f5493b6c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:45.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7fdb42d9d640 1 --2- 172.21.15.97:0/1518953953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413fd60 0x7fdb441401f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb340097d0 tx=0x7fdb3402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97445271aff291a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7fa11e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa118038fe0 0x7fa11802fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7f184b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c0b3c60 0x7f184c1dbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38030/0 (socket says 172.21.15.97:38030) 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7f184b7fe640 1 -- 172.21.15.97:0/4176061961 learned_addr learned my addr 172.21.15.97:0/4176061961 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7f184b7fe640 10 monclient(hunting): get_auth_request con 0x7f184c0b3c60 auth_method 0 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7f184b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7f184b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.390+0000 7f184b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f184bfff640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f184c0b1750 0x7f184c0a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f184b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f184b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f184b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8440143920 0x7f8440148110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38848/0 (socket says 172.21.15.97:38848) 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 1 -- 172.21.15.97:0/550950497 learned_addr learned my addr 172.21.15.97:0/550950497 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 10 monclient(hunting): get_auth_request con 0x7f8440143920 auth_method 0 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7fa11e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa118038fe0 0x7fa11802fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38502/0 (socket says 172.21.15.97:38502) 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7fa11e7fc640 1 -- 172.21.15.97:0/3798491626 learned_addr learned my addr 172.21.15.97:0/3798491626 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7fa11e7fc640 10 monclient(hunting): get_auth_request con 0x7fa118038fe0 auth_method 0 2023-12-07T07:33:45.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7fa11e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7fa11e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7fa11e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.392+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f8444df2640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400af360 0x7f8440148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f184bfff640 10 monclient(hunting): get_auth_request con 0x7f184c0b1750 auth_method 0 2023-12-07T07:33:45.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f184bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f184bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f184bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7fa11e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7fa11e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7fa11e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f1850b0b640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c142e70 0x7f184c1dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f184bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f184bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f184bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f8444df2640 10 monclient(hunting): get_auth_request con 0x7f84400af360 auth_method 0 2023-12-07T07:33:45.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f8444df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f8444df2640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.393+0000 7f8444df2640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 1 -- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 msgr2=0x7fa118165030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa118035640 0x7fa118165030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 1 -- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1180cc600 msgr2=0x7fa11802d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1180cc600 0x7fa11802d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7fa11e7fc640 1 -- 172.21.15.97:0/3798491626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140094d0 con 0x7fa118038fe0 2023-12-07T07:33:45.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7f184bfff640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2023-12-07T07:33:45.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.396+0000 7f184bfff640 1 -- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c0b3c60 msgr2=0x7f184c1dbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f184c0b3c60 0x7f184c1dbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 1 -- 172.21.15.97:0/4176061961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c142e70 msgr2=0x7f184c1dc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c142e70 0x7f184c1dc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 1 monclient: found mon.noname-a 2023-12-07T07:33:45.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.397+0000 7f184bfff640 1 -- 172.21.15.97:0/4176061961 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c009400 con 0x7f184c0b1750 2023-12-07T07:33:45.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.399+0000 7f8444df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.399+0000 7f8444df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.399+0000 7f8444df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 1 -- 172.21.15.97:0/2126005997 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40a6b70 con 0x7ffaec036df0 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: dump: 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.445 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 1 -- 172.21.15.97:0/2126005997 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf40a6cd0 con 0x7ffaec036df0 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 1 -- 172.21.15.97:0/2126005997 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40ae640 con 0x7ffaec036df0 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffafb393640 5 monclient: authenticate success, global_id 4324 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffafb393640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffafb393640 4 set_mon_vals no callback set 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.400+0000 7ffaf938f640 10 monclient: dump: 2023-12-07T07:33:45.446 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 10 monclient: shutdown 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 -- 172.21.15.97:0/2126005997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 msgr2=0x7ffaec0c2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 --2- 172.21.15.97:0/2126005997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 0x7ffaec0c2bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4050140 tx=0x7ffaf40a5d00 comp rx=0 tx=0).stop 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 -- 172.21.15.97:0/2126005997 shutdown_connections 2023-12-07T07:33:45.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 --2- 172.21.15.97:0/2126005997 >> v2:172.21.15.97:3300/0 conn(0x7ffaec0c8f00 0x7ffaec0c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 --2- 172.21.15.97:0/2126005997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec0c3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 --2- 172.21.15.97:0/2126005997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 0x7ffaec0c2bf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 -- 172.21.15.97:0/2126005997 >> 172.21.15.97:0/2126005997 conn(0x7ffaec006390 msgr2=0x7ffaec0067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 -- 172.21.15.97:0/2126005997 shutdown_connections 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.401+0000 7ffafb393640 1 -- 172.21.15.97:0/2126005997 wait complete. 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 10 monclient: monmap: 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.448 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 1 librados: starting msgr at 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 1 librados: starting objecter 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 1 Processor -- start 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 1 -- start start 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 1 librados: setting wanted keys 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 1 librados: calling monclient init 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.402+0000 7ffafb393640 10 monclient: init 2023-12-07T07:33:45.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f5492109640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f547800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 1 -- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440149dc0 msgr2=0x7f84401db760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8440149dc0 0x7f84401db760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 1 -- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8440143920 msgr2=0x7f8440148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8440143920 0x7f8440148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.450 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f8444df2640 1 -- 172.21.15.97:0/550950497 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434009400 con 0x7f84400af360 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f5492109640 10 monclient(hunting): get_auth_request con 0x7f5478005380 auth_method 0 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f5492109640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f5492109640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7f5492109640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec033220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec0355b0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 0x7ffaec0c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec036df0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaec0c8f00 0x7ffaec0307a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec0c8f00 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec030f50 con 0x7ffaec0c8f00 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec030ce0 con 0x7ffaec036df0 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec032b20 con 0x7ffaec0355b0 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.403+0000 7ffafb393640 10 monclient(hunting): authenticate will time out at 1347.782203s 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.404+0000 7f5492109640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.404+0000 7f5492109640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.453 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.404+0000 7f5492109640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:33:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.405+0000 7f184bfff640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f184c0b1750 0x7f184c0a9b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f184000c430 tx=0x7f184000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.405+0000 7f1848ff9640 10 client.?.objecter ms_handle_connect 0x7f184c0b1750 2023-12-07T07:33:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.405+0000 7f1848ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7ffaf9b90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 0x7ffaec0c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f8444df2640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400af360 0x7f8440148f20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f843800cb10 tx=0x7f843800a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7ffaf9b90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 0x7ffaec0c73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38530/0 (socket says 172.21.15.97:38530) 2023-12-07T07:33:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7ffaf9b90640 1 -- 172.21.15.97:0/3904395677 learned_addr learned my addr 172.21.15.97:0/3904395677 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7ffaf9b90640 10 monclient(hunting): get_auth_request con 0x7ffaec036df0 auth_method 0 2023-12-07T07:33:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7ffaf9b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7ffaf9b90640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7ffaf9b90640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 10 monclient(hunting): handle_auth_done global_id 4321 payload 293 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 1 -- 172.21.15.97:0/2565904305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54780047c0 msgr2=0x7f5478004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54780047c0 0x7f5478004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 1 -- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5478003970 msgr2=0x7f5478003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5478003970 0x7f5478003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 1 monclient: found mon.c 2023-12-07T07:33:45.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 1 -- 172.21.15.97:0/2565904305 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5478003520 con 0x7f5478005380 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f5492109640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f843d7fa640 10 client.?.objecter ms_handle_connect 0x7f84400af360 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.407+0000 7f843d7fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.409+0000 7fa11e7fc640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa118038fe0 0x7fa11802fdd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa11000e9b0 tx=0x7fa11000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.409+0000 7fa1077fe640 10 client.?.objecter ms_handle_connect 0x7fa118038fe0 2023-12-07T07:33:45.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.409+0000 7fa1077fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.410+0000 7ffafa391640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec033220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.410+0000 7f5492109640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f547800b980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f548c074c00 tx=0x7f54780057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c4d939380f2da79 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.411+0000 7ffafa391640 10 monclient(hunting): get_auth_request con 0x7ffaec0355b0 auth_method 0 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.411+0000 7ffafa391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.411+0000 7ffafa391640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.411+0000 7ffafa391640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.411+0000 7ffaf9b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.411+0000 7ffaf9b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.411+0000 7ffaf9b90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.412+0000 7ffafa391640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.412+0000 7ffafa391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.412+0000 7ffafa391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafab92640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaec0c8f00 0x7ffaec0307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 1 -- 172.21.15.97:0/3904395677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 msgr2=0x7ffaec0c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaec036df0 0x7ffaec0c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 1 -- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaec0c8f00 msgr2=0x7ffaec0307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaec0c8f00 0x7ffaec0307a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.416+0000 7ffafa391640 1 -- 172.21.15.97:0/3904395677 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec032e00 con 0x7ffaec0355b0 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110009670 con 0x7fa118038fe0 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: dump: 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.461 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11000c9a0 con 0x7fa118038fe0 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 5 monclient: authenticate success, global_id 4327 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 10 monclient: _renew_subs 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa118030310 con 0x7fa118038fe0 2023-12-07T07:33:45.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11000cd50 con 0x7fa118038fe0 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa1077fe640 10 monclient: dump: 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.463 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11d7fa640 4 set_mon_vals no callback set 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 10 monclient: _renew_subs 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11802dc00 con 0x7fa118038fe0 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.417+0000 7fa11ffff640 1 librados: init done 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.418+0000 7fa11ffff640 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.418+0000 7fa11ffff640 15 client.4327.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.418+0000 7fa11ffff640 20 client.4327.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.418+0000 7fa11ffff640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.418+0000 7fa11ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.418+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa1180cca90 con 0x7fa118038fe0 2023-12-07T07:33:45.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.419+0000 7ffafa391640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec033220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4074c70 tx=0x7ffaf40691a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.419+0000 7ffaeaffd640 10 client.?.objecter ms_handle_connect 0x7ffaec0355b0 2023-12-07T07:33:45.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.419+0000 7ffaeaffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.423+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840005ce0 con 0x7f184c0b1750 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.423+0000 7f1848ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f1848ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f1848ff9640 10 monclient: dump: 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.467 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f1848ff9640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f1848ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f1848ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1840005e40 con 0x7f184c0b1750 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 5 monclient: authenticate success, global_id 4290 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 10 monclient: _renew_subs 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c0aa100 con 0x7f184c0b1750 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f1848ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f184a7fc640 4 set_mon_vals no callback set 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 10 monclient: _renew_subs 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c148730 con 0x7f184c0b1750 2023-12-07T07:33:45.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.424+0000 7f185256e640 1 librados: init done 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18400072d0 con 0x7f184c0b1750 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f1848ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f1848ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f1848ff9640 10 monclient: dump: 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.469 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f185256e640 10 client.4290.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f185256e640 15 client.4290.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f185256e640 20 client.4290.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f185256e640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f185256e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.425+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f184c143ea0 con 0x7f184c0b1750 2023-12-07T07:33:45.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843800abf0 con 0x7f84400af360 2023-12-07T07:33:45.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: dump: 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843800ad50 con 0x7f84400af360 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438012620 con 0x7f84400af360 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843d7fa640 10 monclient: dump: 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.473 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 5 monclient: authenticate success, global_id 4330 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 10 monclient: _renew_subs 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8440148650 con 0x7f84400af360 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f843effd640 4 set_mon_vals no callback set 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 10 monclient: _renew_subs 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84401dbe80 con 0x7f84400af360 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.430+0000 7f8447056640 1 librados: init done 2023-12-07T07:33:45.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.431+0000 7f8447056640 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.431+0000 7f8447056640 15 client.4330.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.431+0000 7f8447056640 20 client.4330.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.431+0000 7f8447056640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.431+0000 7f8447056640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.431+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f844006fdc0 con 0x7f84400af360 2023-12-07T07:33:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.443+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa110013470 con 0x7fa118038fe0 2023-12-07T07:33:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.443+0000 7fa1077fe640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa10c06c820 0x7fa10c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.444+0000 7fa11effd640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa10c06c820 0x7fa10c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.444+0000 7fa11effd640 10 monclient: get_auth_request con 0x7fa10c06c820 auth_method 0 2023-12-07T07:33:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.444+0000 7fa11effd640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa10c06c820 0x7fa10c06ece0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa114036000 tx=0x7fa114030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.444+0000 7fa1077fe640 10 client.4327.objecter ms_handle_connect 0x7fa10c06c820 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa1100590f0 con 0x7fa118038fe0 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 10 client.4327.objecter ms_dispatch 0x7fa120000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 3 client.4327.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 3 client.4327.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 10 client.4327.objecter checking command 1 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7fa10c072c00 0x7fa10c075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 20 client.4327.objecter _get_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 20 client.4327.objecter put_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 15 client.4327.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 20 client.4327.objecter _get_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 20 client.4327.objecter get_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 15 client.4327.objecter _session_command_op_assign 2 1 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 20 client.4327.objecter put_session s=0x7fa10c072530 osd=2 4 2023-12-07T07:33:45.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 10 client.4327.objecter _send_command 1 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 --> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa10c0756d0 con 0x7fa10c072c00 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 20 client.4327.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa110054d30 con 0x7fa118038fe0 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa1077fe640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.445+0000 7fa11f7fe640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7fa10c072c00 0x7fa10c075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.446+0000 7fa11f7fe640 10 monclient: get_auth_request con 0x7fa10c072c00 auth_method 0 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.449+0000 7fa11f7fe640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7fa10c072c00 0x7fa10c075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.449+0000 7fa1077fe640 10 client.4327.objecter ms_handle_connect 0x7fa10c072c00 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.453+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1840007430 con 0x7f184c0b1750 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.453+0000 7f1848ff9640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f183806c800 0x7f183806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.453+0000 7f184b7fe640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f183806c800 0x7f183806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.454+0000 7f184b7fe640 10 monclient: get_auth_request con 0x7f183806c800 auth_method 0 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.454+0000 7f1848ff9640 10 client.4290.objecter ms_handle_connect 0x7f183806c800 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.454+0000 7f184b7fe640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f183806c800 0x7f183806ecc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f184c06f380 tx=0x7f183c03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1840058060 con 0x7f184c0b1750 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 10 client.4290.objecter ms_dispatch 0x7f184c000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 3 client.4290.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 3 client.4290.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 10 client.4290.objecter checking command 1 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1838072a70 0x7f1838074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 20 client.4290.objecter _get_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 20 client.4290.objecter put_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 15 client.4290.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 20 client.4290.objecter _get_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 20 client.4290.objecter get_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 15 client.4290.objecter _session_command_op_assign 0 1 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 20 client.4290.objecter put_session s=0x7f1840005e40 osd=0 4 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 10 client.4290.objecter _send_command 1 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1838075540 con 0x7f1838072a70 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 20 client.4290.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18400543a0 con 0x7f184c0b1750 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.457+0000 7f1848ff9640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.458+0000 7f1850b0b640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1838072a70 0x7f1838074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.458+0000 7f1850b0b640 10 monclient: get_auth_request con 0x7f1838072a70 auth_method 0 2023-12-07T07:33:45.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.461+0000 7f1850b0b640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1838072a70 0x7f1838074e90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.461+0000 7f1848ff9640 10 client.4290.objecter ms_handle_connect 0x7f1838072a70 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.468+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8438007070 con 0x7f84400af360 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.468+0000 7f843d7fa640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f842c06c820 0x7f842c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.468+0000 7f843ffff640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f842c06c820 0x7f842c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.468+0000 7f843ffff640 10 monclient: get_auth_request con 0x7f842c06c820 auth_method 0 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.469+0000 7f843d7fa640 10 client.4330.objecter ms_handle_connect 0x7f842c06c820 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.469+0000 7f843ffff640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f842c06c820 0x7f842c06ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8440149810 tx=0x7f8430008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8438058850 con 0x7f84400af360 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 10 client.4330.objecter ms_dispatch 0x7f8440000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 3 client.4330.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 3 client.4330.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 10 client.4330.objecter checking command 1 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f842c072c00 0x7f842c075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 20 client.4330.objecter _get_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 20 client.4330.objecter put_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 15 client.4330.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 20 client.4330.objecter _get_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 20 client.4330.objecter get_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 15 client.4330.objecter _session_command_op_assign 5 1 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 20 client.4330.objecter put_session s=0x7f842c072530 osd=5 4 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 10 client.4330.objecter _send_command 1 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 --> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f842c0756d0 con 0x7f842c072c00 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.470+0000 7f843d7fa640 20 client.4330.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.471+0000 7f84455f3640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f842c072c00 0x7f842c075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.471+0000 7f84455f3640 10 monclient: get_auth_request con 0x7f842c072c00 auth_method 0 2023-12-07T07:33:45.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.474+0000 7f84455f3640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f842c072c00 0x7f842c075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.474+0000 7f843d7fa640 10 client.4330.objecter ms_handle_connect 0x7f842c072c00 2023-12-07T07:33:45.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.498+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== osd.2 v2:172.21.15.97:6802/1673674356 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa10c0756d0 con 0x7fa10c072c00 2023-12-07T07:33:45.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.498+0000 7fa1077fe640 10 client.4327.objecter ms_dispatch 0x7fa120000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.498+0000 7fa1077fe640 10 client.4327.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.498+0000 7fa1077fe640 20 client.4327.objecter put_session s=0x7fa10c072530 osd=2 4 2023-12-07T07:33:45.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.498+0000 7fa1077fe640 15 client.4327.objecter _session_command_op_remove 2 1 2023-12-07T07:33:45.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.511+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== osd.0 v2:172.21.15.97:6818/2798215727 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1838075540 con 0x7f1838072a70 2023-12-07T07:33:45.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.511+0000 7f1848ff9640 10 client.4290.objecter ms_dispatch 0x7f184c000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.511+0000 7f1848ff9640 10 client.4290.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.511+0000 7f1848ff9640 20 client.4290.objecter put_session s=0x7f1840005e40 osd=0 4 2023-12-07T07:33:45.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.511+0000 7f1848ff9640 15 client.4290.objecter _session_command_op_remove 0 1 2023-12-07T07:33:45.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.515+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 <== osd.5 v2:172.21.15.154:6800/4174110211 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f842c0756d0 con 0x7f842c072c00 2023-12-07T07:33:45.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.515+0000 7f843d7fa640 10 client.4330.objecter ms_dispatch 0x7f8440000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.515+0000 7f843d7fa640 10 client.4330.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.515+0000 7f843d7fa640 20 client.4330.objecter put_session s=0x7f842c072530 osd=5 4 2023-12-07T07:33:45.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.515+0000 7f843d7fa640 15 client.4330.objecter _session_command_op_remove 5 1 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 15 client.4327.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 20 client.4327.objecter _get_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 20 client.4327.objecter put_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 20 client.4327.objecter _get_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 15 client.4327.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 20 client.4327.objecter get_session s=0x7fa10c072530 osd=2 3 2023-12-07T07:33:45.520 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 15 client.4327.objecter _session_command_op_assign 2 2 2023-12-07T07:33:45.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 20 client.4327.objecter put_session s=0x7fa10c072530 osd=2 4 2023-12-07T07:33:45.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 10 client.4327.objecter _send_command 2 2023-12-07T07:33:45.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.518+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 --> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa11802fb00 con 0x7fa10c072c00 2023-12-07T07:33:45.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa1077fe640 1 -- 172.21.15.97:0/3798491626 <== osd.2 v2:172.21.15.97:6802/1673674356 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa11802fb00 con 0x7fa10c072c00 2023-12-07T07:33:45.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa1077fe640 10 client.4327.objecter ms_dispatch 0x7fa120000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa1077fe640 10 client.4327.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa1077fe640 20 client.4327.objecter put_session s=0x7fa10c072530 osd=2 4 2023-12-07T07:33:45.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa1077fe640 15 client.4327.objecter _session_command_op_remove 2 2 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 10 librados: watch_flush enter 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 10 librados: watch_flush exit 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 10 client.4327.objecter close_session for osd.2 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7fa10c072c00 msgr2=0x7fa10c075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7fa10c072c00 0x7fa10c075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 20 client.4327.objecter put_session s=0x7fa10c072530 osd=2 1 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 20 client.4327.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.529+0000 7fa11ffff640 10 client.4327.objecter successfully canceled tick 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa10c06c820 msgr2=0x7fa10c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa10c06c820 0x7fa10c06ece0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa114036000 tx=0x7fa114030000 comp rx=0 tx=0).stop 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 10 monclient: shutdown 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa118038fe0 msgr2=0x7fa11802fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 --2- 172.21.15.97:0/3798491626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa118038fe0 0x7fa11802fdd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa11000e9b0 tx=0x7fa11000ee70 comp rx=0 tx=0).stop 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11f7fe640 1 -- 172.21.15.97:0/3798491626 reap_dead start 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 shutdown_connections 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 >> 172.21.15.97:0/3798491626 conn(0x7fa1180065d0 msgr2=0x7fa118004e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 shutdown_connections 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 -- 172.21.15.97:0/3798491626 wait complete. 2023-12-07T07:33:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.530+0000 7fa11ffff640 1 librados: shutdown 2023-12-07T07:33:45.537 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 10 client.4290.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 15 client.4290.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 20 client.4290.objecter _get_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 20 client.4290.objecter put_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 20 client.4290.objecter _get_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 15 client.4290.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 20 client.4290.objecter get_session s=0x7f1840005e40 osd=0 3 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 15 client.4290.objecter _session_command_op_assign 0 2 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 20 client.4290.objecter put_session s=0x7f1840005e40 osd=0 4 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 10 client.4290.objecter _send_command 2 2023-12-07T07:33:45.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.535+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f184c067ac0 con 0x7f1838072a70 2023-12-07T07:33:45.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 15 client.4330.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 20 client.4330.objecter _get_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 20 client.4330.objecter put_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 20 client.4330.objecter _get_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 15 client.4330.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 20 client.4330.objecter get_session s=0x7f842c072530 osd=5 3 2023-12-07T07:33:45.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 15 client.4330.objecter _session_command_op_assign 5 2 2023-12-07T07:33:45.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 20 client.4330.objecter put_session s=0x7f842c072530 osd=5 4 2023-12-07T07:33:45.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 10 client.4330.objecter _send_command 2 2023-12-07T07:33:45.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.539+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 --> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f844006fae0 con 0x7f842c072c00 2023-12-07T07:33:45.543 INFO:teuthology.orchestra.run.smithi097.stdout:73014444037 2023-12-07T07:33:45.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-07T07:33:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.543+0000 7f1848ff9640 1 -- 172.21.15.97:0/4176061961 <== osd.0 v2:172.21.15.97:6818/2798215727 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f184c067ac0 con 0x7f1838072a70 2023-12-07T07:33:45.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.543+0000 7f1848ff9640 10 client.4290.objecter ms_dispatch 0x7f184c000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.543+0000 7f1848ff9640 10 client.4290.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.543+0000 7f1848ff9640 20 client.4290.objecter put_session s=0x7f1840005e40 osd=0 4 2023-12-07T07:33:45.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.543+0000 7f1848ff9640 15 client.4290.objecter _session_command_op_remove 0 2 2023-12-07T07:33:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 10 librados: watch_flush enter 2023-12-07T07:33:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 10 librados: watch_flush exit 2023-12-07T07:33:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 10 client.4290.objecter close_session for osd.0 2023-12-07T07:33:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1838072a70 msgr2=0x7f1838074e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.552 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1838072a70 0x7f1838074e90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 20 client.4290.objecter put_session s=0x7f1840005e40 osd=0 1 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 20 client.4290.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 10 client.4290.objecter successfully canceled tick 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f183806c800 msgr2=0x7f183806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f183806c800 0x7f183806ecc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f184c06f380 tx=0x7f183c03d040 comp rx=0 tx=0).stop 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 10 monclient: shutdown 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f184c0b1750 msgr2=0x7f184c0a9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 --2- 172.21.15.97:0/4176061961 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f184c0b1750 0x7f184c0a9b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f184000c430 tx=0x7f184000c8f0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f1850b0b640 1 -- 172.21.15.97:0/4176061961 reap_dead start 2023-12-07T07:33:45.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 shutdown_connections 2023-12-07T07:33:45.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 >> 172.21.15.97:0/4176061961 conn(0x7f184c0b4a00 msgr2=0x7f184c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 shutdown_connections 2023-12-07T07:33:45.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 -- 172.21.15.97:0/4176061961 wait complete. 2023-12-07T07:33:45.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.544+0000 7f185256e640 1 librados: shutdown 2023-12-07T07:33:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 1 -- 172.21.15.97:0/425832867 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314030710 con 0x7f531c033180 2023-12-07T07:33:45.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: dump: 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 1 -- 172.21.15.97:0/425832867 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531402e050 con 0x7f531c033180 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f532a659640 5 monclient: authenticate success, global_id 4271 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f532a659640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f532a659640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 1 -- 172.21.15.97:0/425832867 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314030a90 con 0x7f531c033180 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f532a659640 4 set_mon_vals no callback set 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7f531bfff640 10 monclient: dump: 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.559 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 1 -- 172.21.15.97:0/4153908151 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e002d040 con 0x7fd4ec14a360 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: dump: 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.560 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.561 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.561 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.561 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 1 -- 172.21.15.97:0/4153908151 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e0031810 con 0x7fd4ec14a360 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 1 -- 172.21.15.97:0/4153908151 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0031ba0 con 0x7fd4ec14a360 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4e959a640 10 monclient: dump: 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.562 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4f0fb0640 5 monclient: authenticate success, global_id 4268 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4f0fb0640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.554+0000 7fd4f0fb0640 4 set_mon_vals no callback set 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 10 monclient: shutdown 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 1 -- 172.21.15.97:0/425832867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c033180 msgr2=0x7f531c0d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 1 --2- 172.21.15.97:0/425832867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c033180 0x7f531c0d1270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f53140097d0 tx=0x7f531402b6f0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 1 -- 172.21.15.97:0/425832867 shutdown_connections 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 1 --2- 172.21.15.97:0/425832867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c033180 0x7f531c0d1270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 1 -- 172.21.15.97:0/425832867 >> 172.21.15.97:0/425832867 conn(0x7f531c005ea0 msgr2=0x7f531c0062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 10 monclient: shutdown 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/4153908151 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec14a360 msgr2=0x7fd4ec151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 1 --2- 172.21.15.97:0/4153908151 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec14a360 0x7fd4ec151a70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00097d0 tx=0x7fd4e002b8b0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 1 -- 172.21.15.97:0/425832867 shutdown_connections 2023-12-07T07:33:45.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7f532a659640 1 -- 172.21.15.97:0/425832867 wait complete. 2023-12-07T07:33:45.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/4153908151 shutdown_connections 2023-12-07T07:33:45.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 1 --2- 172.21.15.97:0/4153908151 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec14a360 0x7fd4ec151a70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/4153908151 >> 172.21.15.97:0/4153908151 conn(0x7fd4ec0b01e0 msgr2=0x7fd4ec0b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/4153908151 shutdown_connections 2023-12-07T07:33:45.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.555+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/4153908151 wait complete. 2023-12-07T07:33:45.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.556+0000 7f532a659640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.556+0000 7f532a659640 10 monclient: monmap: 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.567 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.556+0000 7fd4f0fb0640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.556+0000 7fd4f0fb0640 10 monclient: monmap: 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.568 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.556+0000 7f532a659640 1 librados: starting msgr at 2023-12-07T07:33:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.556+0000 7f532a659640 1 librados: starting objecter 2023-12-07T07:33:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.556+0000 7f532a659640 1 Processor -- start 2023-12-07T07:33:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 -- start start 2023-12-07T07:33:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 librados: setting wanted keys 2023-12-07T07:33:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 librados: calling monclient init 2023-12-07T07:33:45.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient: init 2023-12-07T07:33:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7fd4f0fb0640 1 librados: starting msgr at 2023-12-07T07:33:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7fd4f0fb0640 1 librados: starting objecter 2023-12-07T07:33:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7fd4f0fb0640 1 Processor -- start 2023-12-07T07:33:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7fd4f0fb0640 1 -- start start 2023-12-07T07:33:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7fd4f0fb0640 1 librados: setting wanted keys 2023-12-07T07:33:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7fd4f0fb0640 1 librados: calling monclient init 2023-12-07T07:33:45.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7fd4f0fb0640 10 monclient: init 2023-12-07T07:33:45.571 INFO:teuthology.orchestra.run.smithi097.stdout:68719476741 2023-12-07T07:33:45.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-07T07:33:45.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c006de0 0x7f531c02f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): picked mon.noname-c con 0x7f531c006de0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f531c033180 0x7f531c0290f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): picked mon.noname-a con 0x7f531c033180 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f531c0c8c60 0x7f531c02acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): picked mon.noname-b con 0x7f531c0c8c60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c02eef0 con 0x7f531c033180 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c02f8e0 con 0x7f531c0c8c60 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c02a550 con 0x7f531c006de0 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.557+0000 7f532a659640 10 monclient(hunting): authenticate will time out at 1347.936112s 2023-12-07T07:33:45.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:45.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec06f2b0 0x7fd4ec0a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): picked mon.noname-c con 0x7fd4ec06f2b0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4ec0af360 0x7fd4ec0ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ec0af360 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ec14a360 0x7fd4ec0ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ec14a360 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0ab8f0 con 0x7fd4ec14a360 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0a8590 con 0x7fd4ec0af360 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0abc50 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fd4f0fb0640 10 monclient(hunting): authenticate will time out at 1347.936551s 2023-12-07T07:33:45.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7f843d7fa640 1 -- 172.21.15.97:0/550950497 <== osd.5 v2:172.21.15.154:6800/4174110211 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8434031a40 con 0x7f842c072c00 2023-12-07T07:33:45.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7f843d7fa640 10 client.4330.objecter ms_dispatch 0x7f8440000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7f843d7fa640 10 client.4330.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7f843d7fa640 20 client.4330.objecter put_session s=0x7f842c072530 osd=5 4 2023-12-07T07:33:45.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7f843d7fa640 15 client.4330.objecter _session_command_op_remove 5 2 2023-12-07T07:33:45.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 1 -- 172.21.15.97:0/1518953953 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb340316f0 con 0x7fdb4413fd60 2023-12-07T07:33:45.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: dump: 2023-12-07T07:33:45.577 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.577 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 1 -- 172.21.15.97:0/1518953953 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb34031850 con 0x7fdb4413fd60 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 1 -- 172.21.15.97:0/1518953953 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34031be0 con 0x7fdb4413fd60 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb4159a640 10 monclient: dump: 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.579 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.580 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.580 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.580 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.580 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb48ad0640 5 monclient: authenticate success, global_id 4277 2023-12-07T07:33:45.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb48ad0640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.558+0000 7fdb48ad0640 4 set_mon_vals no callback set 2023-12-07T07:33:45.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 10 librados: watch_flush enter 2023-12-07T07:33:45.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 10 librados: watch_flush exit 2023-12-07T07:33:45.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 10 client.4330.objecter close_session for osd.5 2023-12-07T07:33:45.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f842c072c00 msgr2=0x7f842c075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f842c072c00 0x7f842c075020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 20 client.4330.objecter put_session s=0x7f842c072530 osd=5 1 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 20 client.4330.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 10 client.4330.objecter successfully canceled tick 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f842c06c820 msgr2=0x7f842c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f842c06c820 0x7f842c06ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8440149810 tx=0x7f8430008040 comp rx=0 tx=0).stop 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 10 monclient: shutdown 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 20 monclient: shutdown canceling and discarding version request 1 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400af360 msgr2=0x7f8440148f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 --2- 172.21.15.97:0/550950497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400af360 0x7f8440148f20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f843800cb10 tx=0x7f843800a060 comp rx=0 tx=0).stop 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f84455f3640 1 -- 172.21.15.97:0/550950497 reap_dead start 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 shutdown_connections 2023-12-07T07:33:45.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 >> 172.21.15.97:0/550950497 conn(0x7f84400b01e0 msgr2=0x7f8440142fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 10 monclient: shutdown 2023-12-07T07:33:45.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1518953953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413fd60 msgr2=0x7fdb441401f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 1 --2- 172.21.15.97:0/1518953953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413fd60 0x7fdb441401f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb340097d0 tx=0x7fdb3402b790 comp rx=0 tx=0).stop 2023-12-07T07:33:45.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 shutdown_connections 2023-12-07T07:33:45.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 -- 172.21.15.97:0/550950497 wait complete. 2023-12-07T07:33:45.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7f8447056640 1 librados: shutdown 2023-12-07T07:33:45.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1518953953 shutdown_connections 2023-12-07T07:33:45.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 1 --2- 172.21.15.97:0/1518953953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413fd60 0x7fdb441401f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1518953953 >> 172.21.15.97:0/1518953953 conn(0x7fdb440b2310 msgr2=0x7fdb440b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1518953953 shutdown_connections 2023-12-07T07:33:45.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.559+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1518953953 wait complete. 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.560+0000 7fdb48ad0640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.560+0000 7fdb48ad0640 10 monclient: monmap: 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.561+0000 7fdb48ad0640 1 librados: starting msgr at 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.561+0000 7fdb48ad0640 1 librados: starting objecter 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.561+0000 7fdb48ad0640 1 Processor -- start 2023-12-07T07:33:45.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.561+0000 7fdb48ad0640 1 -- start start 2023-12-07T07:33:45.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.561+0000 7fdb48ad0640 1 librados: setting wanted keys 2023-12-07T07:33:45.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.561+0000 7fdb48ad0640 1 librados: calling monclient init 2023-12-07T07:33:45.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.561+0000 7fdb48ad0640 10 monclient: init 2023-12-07T07:33:45.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c0a77a0 con 0x7f5478005380 2023-12-07T07:33:45.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:33:45.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb440b1490 0x7fdb44134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 10 monclient: dump: 2023-12-07T07:33:45.588 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.588 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.588 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.588 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): picked mon.noname-a con 0x7fdb440b1490 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c0a7900 con 0x7f5478005380 2023-12-07T07:33:45.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413cf10 0x7fdb44137370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5493b6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:45.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 1 -- 172.21.15.97:0/2565904305 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c0a7cb0 con 0x7f5478005380 2023-12-07T07:33:45.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5493b6c640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5493b6c640 4 set_mon_vals no callback set 2023-12-07T07:33:45.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): picked mon.noname-c con 0x7fdb4413cf10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb4413fd60 0x7fdb44141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4413fd60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb44137010 con 0x7fdb440b1490 2023-12-07T07:33:45.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb441438a0 con 0x7fdb4413fd60 2023-12-07T07:33:45.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb44142240 con 0x7fdb4413cf10 2023-12-07T07:33:45.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7fdb48ad0640 10 monclient(hunting): authenticate will time out at 1347.940829s 2023-12-07T07:33:45.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.562+0000 7f5490906640 10 monclient: dump: 2023-12-07T07:33:45.594 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.594 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.594 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 10 monclient: shutdown 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 1 -- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 msgr2=0x7f547800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 1 --2- 172.21.15.97:0/2565904305 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f547800b980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f548c074c00 tx=0x7f54780057f0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5492109640 1 -- 172.21.15.97:0/2565904305 reap_dead start 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 1 -- 172.21.15.97:0/2565904305 shutdown_connections 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 1 -- 172.21.15.97:0/2565904305 >> 172.21.15.97:0/2565904305 conn(0x7f54840065d0 msgr2=0x7f54840069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 1 -- 172.21.15.97:0/2565904305 shutdown_connections 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.563+0000 7f5493b6c640 1 -- 172.21.15.97:0/2565904305 wait complete. 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 10 monclient: monmap: 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 1 librados: starting msgr at 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 1 librados: starting objecter 2023-12-07T07:33:45.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 1 Processor -- start 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 1 -- start start 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 1 librados: setting wanted keys 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 1 librados: calling monclient init 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.564+0000 7f5493b6c640 10 monclient: init 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f54840cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): picked mon.noname-c con 0x7f5478005380 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f548415fc80 0x7f5484160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): picked mon.noname-b con 0x7f548415fc80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54840a5020 0x7f548415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): picked mon.noname-a con 0x7f54840a5020 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5484160b70 con 0x7f54840a5020 2023-12-07T07:33:45.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.598 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5484160110 con 0x7f548415fc80 2023-12-07T07:33:45.598 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.598 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548415ccc0 con 0x7f5478005380 2023-12-07T07:33:45.598 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.598 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.565+0000 7f5493b6c640 10 monclient(hunting): authenticate will time out at 1347.943976s 2023-12-07T07:33:45.599 INFO:teuthology.orchestra.run.smithi097.stdout:55834574853 2023-12-07T07:33:45.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-07T07:33:45.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40ba650 con 0x7ffaec0355b0 2023-12-07T07:33:45.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: dump: 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf40ba7b0 con 0x7ffaec0355b0 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40bab20 con 0x7ffaec0355b0 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaeaffd640 10 monclient: dump: 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.602 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.575+0000 7ffaf8b8e640 4 set_mon_vals no callback set 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 5 monclient: authenticate success, global_id 4283 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 10 monclient: _renew_subs 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec031490 con 0x7ffaec0355b0 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 10 monclient: _renew_subs 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec031890 con 0x7ffaec0355b0 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.576+0000 7ffafb393640 1 librados: init done 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.577+0000 7ffafb393640 10 client.4283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.577+0000 7ffafb393640 15 client.4283.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.577+0000 7ffafb393640 20 client.4283.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.577+0000 7ffafb393640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.577+0000 7ffafb393640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.577+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffaec0c9ae0 con 0x7ffaec0355b0 2023-12-07T07:33:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.579+0000 7f5329657640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c006de0 0x7f531c02f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.579+0000 7f5329657640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c006de0 0x7f531c02f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38084/0 (socket says 172.21.15.97:38084) 2023-12-07T07:33:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.579+0000 7f5329657640 1 -- 172.21.15.97:0/1917151265 learned_addr learned my addr 172.21.15.97:0/1917151265 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.579+0000 7f5329657640 10 monclient(hunting): get_auth_request con 0x7f531c006de0 auth_method 0 2023-12-07T07:33:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.579+0000 7f5329657640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.579+0000 7f5329657640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.579+0000 7f5329657640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.580+0000 7fd4ea59c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec06f2b0 0x7fd4ec0a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.580+0000 7f5329657640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.580+0000 7f5329657640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.580+0000 7f5329657640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 1 -- 172.21.15.97:0/1917151265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f531c0c8c60 msgr2=0x7f531c02acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f531c0c8c60 0x7f531c02acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 1 -- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f531c033180 msgr2=0x7f531c0290f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f531c033180 0x7f531c0290f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.583+0000 7f5329657640 1 -- 172.21.15.97:0/1917151265 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5314009400 con 0x7f531c006de0 2023-12-07T07:33:45.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.585+0000 7fd4ea59c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec06f2b0 0x7fd4ec0a9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38100/0 (socket says 172.21.15.97:38100) 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.585+0000 7fd4ea59c640 1 -- 172.21.15.97:0/3303074175 learned_addr learned my addr 172.21.15.97:0/3303074175 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.585+0000 7fd4ea59c640 10 monclient(hunting): get_auth_request con 0x7fd4ec06f2b0 auth_method 0 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.585+0000 7fd4ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.585+0000 7fd4ea59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.585+0000 7fd4ea59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.587+0000 7f5329657640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c006de0 0x7f531c02f3a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5324000b60 tx=0x7f5324062060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.587+0000 7f5319ffb640 10 client.?.objecter ms_handle_connect 0x7f531c006de0 2023-12-07T07:33:45.607 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.587+0000 7f5319ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.589+0000 7fdb41d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413cf10 0x7fdb44137370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.590+0000 7fd4ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.590+0000 7fd4ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.590+0000 7fd4ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.591+0000 7fdb41d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413cf10 0x7fdb44137370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38112/0 (socket says 172.21.15.97:38112) 2023-12-07T07:33:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.591+0000 7fdb41d9b640 1 -- 172.21.15.97:0/1694726225 learned_addr learned my addr 172.21.15.97:0/1694726225 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.591+0000 7fdb41d9b640 10 monclient(hunting): get_auth_request con 0x7fdb4413cf10 auth_method 0 2023-12-07T07:33:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.591+0000 7fdb41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.591+0000 7fdb41d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.591+0000 7fdb41d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.591+0000 7f5491908640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f54840cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 1 -- 172.21.15.97:0/3303074175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4ec0af360 msgr2=0x7fd4ec0ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4ec0af360 0x7fd4ec0ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 1 -- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ec14a360 msgr2=0x7fd4ec0ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ec14a360 0x7fd4ec0ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.595+0000 7fd4ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.596+0000 7fd4ea59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.596+0000 7fd4ea59c640 1 -- 172.21.15.97:0/3303074175 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e0009400 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7fdb41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7fdb41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7fdb41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7f5491908640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f54840cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38122/0 (socket says 172.21.15.97:38122) 2023-12-07T07:33:45.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7f5491908640 1 -- 172.21.15.97:0/1218731073 learned_addr learned my addr 172.21.15.97:0/1218731073 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7f5491908640 10 monclient(hunting): get_auth_request con 0x7f5478005380 auth_method 0 2023-12-07T07:33:45.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7f5491908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7f5491908640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.598+0000 7f5491908640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb42d9d640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb4413fd60 0x7fdb44141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 1 -- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb4413fd60 msgr2=0x7fdb44141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb4413fd60 0x7fdb44141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 1 -- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb440b1490 msgr2=0x7fdb44134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb440b1490 0x7fdb44134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.602+0000 7fdb41d9b640 1 -- 172.21.15.97:0/1694726225 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34009400 con 0x7fdb4413cf10 2023-12-07T07:33:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.604+0000 7f5491908640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.604+0000 7f5491908640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.604+0000 7f5491908640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.604+0000 7f5491107640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f548415fc80 0x7f5484160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7fd4ea59c640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec06f2b0 0x7fd4ec0a9f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400c850 tx=0x7fd4e400cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7fd4daffd640 10 client.?.objecter ms_handle_connect 0x7fd4ec06f2b0 2023-12-07T07:33:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7fd4daffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7f5491107640 10 monclient(hunting): get_auth_request con 0x7f548415fc80 auth_method 0 2023-12-07T07:33:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7f5491107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7f5491107640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7f5491107640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7f5491107640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7f5491107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.605+0000 7f5491107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.608+0000 7f5491107640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.608+0000 7f5491107640 1 -- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 msgr2=0x7f54840cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.608+0000 7f5491107640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5478005380 0x7f54840cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.608+0000 7f5491107640 1 -- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54840a5020 msgr2=0x7f548415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.608+0000 7f5491107640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54840a5020 0x7f548415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.609+0000 7f5491107640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.609+0000 7f5491107640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.609+0000 7f5491107640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.609+0000 7f5491107640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.609+0000 7f5491107640 1 -- 172.21.15.97:0/1218731073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54840c3ea0 con 0x7f548415fc80 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.610+0000 7f5491107640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f548415fc80 0x7f5484160630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f547c00afd0 tx=0x7f547c00c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.610+0000 7f54827fc640 10 client.?.objecter ms_handle_connect 0x7f548415fc80 2023-12-07T07:33:45.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.610+0000 7f54827fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.612+0000 7fdb41d9b640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413cf10 0x7fdb44137370 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c530 tx=0x7fdb3800c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.613+0000 7fdb32ffd640 10 client.?.objecter ms_handle_connect 0x7fdb4413cf10 2023-12-07T07:33:45.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.613+0000 7fdb32ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.617+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c011800 con 0x7f548415fc80 2023-12-07T07:33:45.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: dump: 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c011980 con 0x7f548415fc80 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c011d30 con 0x7f548415fc80 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f54827fc640 10 monclient: dump: 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.622 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5483fff640 4 set_mon_vals no callback set 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 5 monclient: authenticate success, global_id 4336 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 10 monclient: _renew_subs 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548402d4e0 con 0x7f548415fc80 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 10 monclient: _renew_subs 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548402d940 con 0x7f548415fc80 2023-12-07T07:33:45.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.618+0000 7f5493b6c640 1 librados: init done 2023-12-07T07:33:45.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.619+0000 7f5493b6c640 10 client.4336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.619+0000 7f5493b6c640 15 client.4336.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.619+0000 7f5493b6c640 20 client.4336.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.619+0000 7f5493b6c640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.619+0000 7f5493b6c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.619+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54840085f0 con 0x7f548415fc80 2023-12-07T07:33:45.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.638+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f547c0233e0 con 0x7f548415fc80 2023-12-07T07:33:45.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.638+0000 7f54827fc640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f547006c820 0x7f547006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f5491908640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f547006c820 0x7f547006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f5491908640 10 monclient: get_auth_request con 0x7f547006c820 auth_method 0 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f5491908640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f547006c820 0x7f547006ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5488000fb0 tx=0x7f5488001a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f54827fc640 10 client.4336.objecter ms_handle_connect 0x7f547006c820 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f547c058810 con 0x7f548415fc80 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f54827fc640 10 client.4336.objecter ms_dispatch 0x7f548c000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f54827fc640 3 client.4336.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.639+0000 7f54827fc640 3 client.4336.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 10 client.4336.objecter checking command 1 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f5470072c00 0x7f5470075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 20 client.4336.objecter _get_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 20 client.4336.objecter put_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 15 client.4336.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 20 client.4336.objecter _get_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 20 client.4336.objecter get_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 15 client.4336.objecter _session_command_op_assign 1 1 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 20 client.4336.objecter put_session s=0x7f5470072530 osd=1 4 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 10 client.4336.objecter _send_command 1 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 --> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54700756d0 con 0x7f5470072c00 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 20 client.4336.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f547c054b50 con 0x7f548415fc80 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.640+0000 7f54827fc640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:33:45.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.641+0000 7f5492109640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f5470072c00 0x7f5470075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.644 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.641+0000 7f5492109640 10 monclient: get_auth_request con 0x7f5470072c00 auth_method 0 2023-12-07T07:33:45.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.645+0000 7f5492109640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f5470072c00 0x7f5470075020 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.645+0000 7f54827fc640 10 client.4336.objecter ms_handle_connect 0x7f5470072c00 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.658+0000 7feb7f221640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.658+0000 7feb7f221640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.658+0000 7feb7f221640 10 monclient: monmap: 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.659855+0000 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.659855+0000 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.662 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.662 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.659+0000 7feb7f221640 1 Processor -- start 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.659+0000 7feb7f221640 1 -- start start 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.659+0000 7feb7f221640 10 monclient: init 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feb780a8a30 0x7feb781424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient(hunting): picked mon.noname-c con 0x7feb780a8a30 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 0x7feb78142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient(hunting): picked mon.noname-b con 0x7feb780a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb781454a0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb780aac10 con 0x7feb781454a0 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb780aaa10 con 0x7feb780a8a30 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb780aaef0 con 0x7feb780a8e10 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7f221640 10 monclient(hunting): authenticate will time out at 1348.038874s 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.660+0000 7feb7cfbd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feb780a8a30 0x7feb781424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7d7be640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feb781454a0 0x7feb78145930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41242/0 (socket says 172.21.15.97:41242) 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7d7be640 1 -- 172.21.15.97:0/456061770 learned_addr learned my addr 172.21.15.97:0/456061770 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7cfbd640 10 monclient(hunting): get_auth_request con 0x7feb780a8a30 auth_method 0 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7cfbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7cfbd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7cfbd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7cfbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7cfbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.661+0000 7feb7cfbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb777fe640 1 -- 172.21.15.97:0/456061770 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583469384 0 0) 0x7feb780aac10 con 0x7feb781454a0 2023-12-07T07:33:45.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb777fe640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb777fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb777fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb777fe640 10 monclient(hunting): my global_id is 4293 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb777fe640 1 -- 172.21.15.97:0/456061770 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb68003520 con 0x7feb781454a0 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 1 -- 172.21.15.97:0/456061770 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 msgr2=0x7feb78142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 1 --2- 172.21.15.97:0/456061770 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 0x7feb78142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 1 -- 172.21.15.97:0/456061770 >> v1:172.21.15.97:6789/0 conn(0x7feb781454a0 legacy=0x7feb78145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.665+0000 7feb7cfbd640 1 -- 172.21.15.97:0/456061770 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb780a97f0 con 0x7feb780a8a30 2023-12-07T07:33:45.669 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.667+0000 7feb7cfbd640 1 --2- 172.21.15.97:0/456061770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb780a8a30 0x7feb781424e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0097d0 tx=0x7feb6c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f774b51ac06cfac5 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 1 -- 172.21.15.97:0/456061770 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c031800 con 0x7feb780a8a30 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: dump: 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.673 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 1 -- 172.21.15.97:0/456061770 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c031960 con 0x7feb780a8a30 2023-12-07T07:33:45.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 1 -- 172.21.15.97:0/456061770 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c031800 con 0x7feb780a8a30 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb777fe640 10 monclient: dump: 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.676 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb7f221640 5 monclient: authenticate success, global_id 4339 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb7f221640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.671+0000 7feb7f221640 4 set_mon_vals no callback set 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 10 monclient: shutdown 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 -- 172.21.15.97:0/456061770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb780a8a30 msgr2=0x7feb781424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 --2- 172.21.15.97:0/456061770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb780a8a30 0x7feb781424e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0097d0 tx=0x7feb6c02b8a0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 -- 172.21.15.97:0/456061770 shutdown_connections 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 --2- 172.21.15.97:0/456061770 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 0x7feb78142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 --2- 172.21.15.97:0/456061770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb780a8a30 0x7feb781424e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 -- 172.21.15.97:0/456061770 >> 172.21.15.97:0/456061770 conn(0x7feb78061a30 msgr2=0x7feb780b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 -- 172.21.15.97:0/456061770 shutdown_connections 2023-12-07T07:33:45.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.672+0000 7feb7f221640 1 -- 172.21.15.97:0/456061770 wait complete. 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 10 monclient: monmap: 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 1 librados: starting msgr at 2023-12-07T07:33:45.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 1 librados: starting objecter 2023-12-07T07:33:45.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 1 Processor -- start 2023-12-07T07:33:45.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 1 -- start start 2023-12-07T07:33:45.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 1 librados: setting wanted keys 2023-12-07T07:33:45.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 1 librados: calling monclient init 2023-12-07T07:33:45.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.673+0000 7feb7f221640 10 monclient: init 2023-12-07T07:33:45.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:33:45.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb780a8a30 0x7feb780af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): picked mon.noname-a con 0x7feb780a8a30 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 0x7feb780b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): picked mon.noname-c con 0x7feb780a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb781454a0 0x7feb7813b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): picked mon.noname-b con 0x7feb781454a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780ad640 con 0x7feb780a8a30 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780af8f0 con 0x7feb781454a0 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb780ae890 con 0x7feb780a8e10 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.674+0000 7feb7f221640 10 monclient(hunting): authenticate will time out at 1348.052886s 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb781454a0 0x7feb7813b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb781454a0 0x7feb7813b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38608/0 (socket says 172.21.15.97:38608) 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 1 -- 172.21.15.97:0/2422480110 learned_addr learned my addr 172.21.15.97:0/2422480110 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 10 monclient(hunting): get_auth_request con 0x7feb781454a0 auth_method 0 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7cfbd640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb780a8a30 0x7feb780af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.675+0000 7feb7d7be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.676+0000 7feb7cfbd640 10 monclient(hunting): get_auth_request con 0x7feb780a8a30 auth_method 0 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.676+0000 7feb7cfbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.676+0000 7feb7cfbd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.676+0000 7feb7cfbd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.676+0000 7feb7cfbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.676+0000 7feb7cfbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.676+0000 7feb7cfbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 1 -- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 msgr2=0x7feb780b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 0x7feb780b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 1 -- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb780a8a30 msgr2=0x7feb780af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb780a8a30 0x7feb780af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.678+0000 7feb7d7be640 1 -- 172.21.15.97:0/2422480110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c009400 con 0x7feb781454a0 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.680+0000 7feb7d7be640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb781454a0 0x7feb7813b670 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb7000ca20 tx=0x7feb7000cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.680+0000 7feb757fa640 10 client.?.objecter ms_handle_connect 0x7feb781454a0 2023-12-07T07:33:45.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.680+0000 7feb757fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532406cc40 con 0x7f531c006de0 2023-12-07T07:33:45.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f5319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f5319ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f5319ffb640 10 monclient: dump: 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f5319ffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f5319ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f5319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f532a659640 5 monclient: authenticate success, global_id 4286 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f532a659640 10 monclient: _renew_subs 2023-12-07T07:33:45.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f532a659640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.681+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c0be5f0 con 0x7f531c006de0 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.682+0000 7f532a659640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.682+0000 7f532a659640 10 monclient: _renew_subs 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.682+0000 7f532a659640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.682+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c02fb40 con 0x7f531c006de0 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.682+0000 7f532a659640 1 librados: init done 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.683+0000 7f532a659640 10 client.4286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.683+0000 7f532a659640 15 client.4286.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.683+0000 7f532a659640 20 client.4286.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.683+0000 7f532a659640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.683+0000 7f532a659640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.683+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f531c0c5970 con 0x7f531c006de0 2023-12-07T07:33:45.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 1 -- 172.21.15.97:0/2422480110 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7000abd0 con 0x7feb781454a0 2023-12-07T07:33:45.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 10 monclient: dump: 2023-12-07T07:33:45.690 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 1 -- 172.21.15.97:0/2422480110 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7000ad30 con 0x7feb781454a0 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb7f221640 5 monclient: authenticate success, global_id 4342 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb7f221640 10 monclient: _renew_subs 2023-12-07T07:33:45.691 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb7f221640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb780afd40 con 0x7feb781454a0 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb7f221640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb757fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.684+0000 7feb76ffd640 4 set_mon_vals no callback set 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb757fa640 1 -- 172.21.15.97:0/2422480110 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70012620 con 0x7feb781454a0 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb757fa640 10 monclient: dump: 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb7f221640 10 monclient: _renew_subs 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb7f221640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7813c370 con 0x7feb781454a0 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.685+0000 7feb7f221640 1 librados: init done 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.686+0000 7feb7f221640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.686+0000 7feb7f221640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.686+0000 7feb7f221640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:45.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.686+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb78067ac0 con 0x7feb781454a0 2023-12-07T07:33:45.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.701+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== osd.1 v2:172.21.15.97:6826/138440183 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f54700756d0 con 0x7f5470072c00 2023-12-07T07:33:45.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.701+0000 7f54827fc640 10 client.4336.objecter ms_dispatch 0x7f548c000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.701+0000 7f54827fc640 10 client.4336.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.701+0000 7f54827fc640 20 client.4336.objecter put_session s=0x7f5470072530 osd=1 4 2023-12-07T07:33:45.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.701+0000 7f54827fc640 15 client.4336.objecter _session_command_op_remove 1 1 2023-12-07T07:33:45.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.702+0000 7feb757fa640 1 -- 172.21.15.97:0/2422480110 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feb70007070 con 0x7feb781454a0 2023-12-07T07:33:45.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.703+0000 7feb757fa640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feb5c06c7d0 0x7feb5c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.703+0000 7feb7cfbd640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feb5c06c7d0 0x7feb5c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.703+0000 7feb7cfbd640 10 monclient: get_auth_request con 0x7feb5c06c7d0 auth_method 0 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532406cda0 con 0x7f531c006de0 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7f5319ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324093670 con 0x7f531c006de0 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7f5319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7f531b7fe640 4 set_mon_vals no callback set 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7f5319ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7f5319ffb640 10 monclient: dump: 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.706 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7feb7cfbd640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feb5c06c7d0 0x7feb5c06ec90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feb6c036000 tx=0x7feb6c03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.707 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.704+0000 7feb757fa640 10 client.4342.objecter ms_handle_connect 0x7feb5c06c7d0 2023-12-07T07:33:45.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 10 client.4336.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 15 client.4336.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 20 client.4336.objecter _get_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 20 client.4336.objecter put_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 20 client.4336.objecter _get_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 15 client.4336.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 20 client.4336.objecter get_session s=0x7f5470072530 osd=1 3 2023-12-07T07:33:45.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 15 client.4336.objecter _session_command_op_assign 1 2 2023-12-07T07:33:45.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 20 client.4336.objecter put_session s=0x7f5470072530 osd=1 4 2023-12-07T07:33:45.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 10 client.4336.objecter _send_command 2 2023-12-07T07:33:45.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.716+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 --> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54840056d0 con 0x7f5470072c00 2023-12-07T07:33:45.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4003b70 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 10 monclient: dump: 2023-12-07T07:33:45.720 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.720 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e4003cd0 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4f0fb0640 5 monclient: authenticate success, global_id 4289 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4f0fb0640 10 monclient: _renew_subs 2023-12-07T07:33:45.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4f0fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.717+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec0ad420 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4f0fb0640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4f0fb0640 10 monclient: _renew_subs 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4f0fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec0aa620 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4f0fb0640 1 librados: init done 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4daffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e400aa10 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4daffd640 10 monclient: dump: 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.718+0000 7fd4e8d99640 4 set_mon_vals no callback set 2023-12-07T07:33:45.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.719+0000 7fd4f0fb0640 10 client.4289.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.719+0000 7fd4f0fb0640 15 client.4289.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.719+0000 7fd4f0fb0640 20 client.4289.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.719+0000 7fd4f0fb0640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.719+0000 7fd4f0fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.719+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4d00028a0 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.720+0000 7f92b676c640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.720+0000 7f92b676c640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.720+0000 7f92b676c640 10 monclient: monmap: 2023-12-07T07:33:45.724 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.721654+0000 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.721654+0000 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.725 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.720+0000 7f31934a2640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.720+0000 7f31934a2640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.721+0000 7f31934a2640 10 monclient: monmap: 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.722039+0000 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.722039+0000 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.726 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.727 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.721+0000 7f92b676c640 1 Processor -- start 2023-12-07T07:33:45.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.721+0000 7f92b676c640 1 -- start start 2023-12-07T07:33:45.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.721+0000 7f31934a2640 1 Processor -- start 2023-12-07T07:33:45.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.721+0000 7f92b676c640 10 monclient: init 2023-12-07T07:33:45.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.721+0000 7f31934a2640 1 -- start start 2023-12-07T07:33:45.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.721+0000 7f31934a2640 10 monclient: init 2023-12-07T07:33:45.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-12-07T07:33:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 0x7f92b01424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 10 monclient(hunting): picked mon.noname-b con 0x7f92b00a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f92b00a8e10 0x7f92b0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-12-07T07:33:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f318c0a8a30 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:45.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f318c142bf0 0x7f318c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient(hunting): picked mon.noname-a con 0x7f318c142bf0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:45.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 10 monclient(hunting): picked mon.noname-a con 0x7f92b00a8e10 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:45.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f318c0a8e10 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c0aac10 con 0x7f318c0a8e10 2023-12-07T07:33:45.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c0aaef0 con 0x7f318c0a8a30 2023-12-07T07:33:45.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318c0aaa10 con 0x7f318c142bf0 2023-12-07T07:33:45.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f31934a2640 10 monclient(hunting): authenticate will time out at 1348.100978s 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f92b01454a0 0x7f92b0145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 10 monclient(hunting): picked mon.noname-c con 0x7f92b01454a0 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.722+0000 7f92b676c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92b676c640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b00aaa10 con 0x7f92b00a8e10 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92b676c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92b676c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b00aac10 con 0x7f92b01454a0 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92b676c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92b676c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00aaef0 con 0x7f92b00a8a30 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92b676c640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92b676c640 10 monclient(hunting): authenticate will time out at 1348.101317s 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f92af7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f92b00a8e10 0x7f92b0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f3191a3f640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f318c0a8e10 0x7f318c13f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41244/0 (socket says 172.21.15.97:41244) 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f3191a3f640 1 -- 172.21.15.97:0/3314475777 learned_addr learned my addr 172.21.15.97:0/3314475777 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f54827fc640 1 -- 172.21.15.97:0/1218731073 <== osd.1 v2:172.21.15.97:6826/138440183 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f54840056d0 con 0x7f5470072c00 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f54827fc640 10 client.4336.objecter ms_dispatch 0x7f548c000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f54827fc640 10 client.4336.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f54827fc640 20 client.4336.objecter put_session s=0x7f5470072530 osd=1 4 2023-12-07T07:33:45.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.723+0000 7f54827fc640 15 client.4336.objecter _session_command_op_remove 1 2 2023-12-07T07:33:45.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f92af7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f92b00a8e10 0x7f92b0142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38928/0 (socket says 172.21.15.97:38928) 2023-12-07T07:33:45.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f92af7fe640 1 -- 172.21.15.97:0/3952028911 learned_addr learned my addr 172.21.15.97:0/3952028911 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f92af7fe640 10 monclient(hunting): get_auth_request con 0x7f92b00a8e10 auth_method 0 2023-12-07T07:33:45.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f92af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 10 librados: watch_flush enter 2023-12-07T07:33:45.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 10 librados: watch_flush exit 2023-12-07T07:33:45.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 10 client.4336.objecter close_session for osd.1 2023-12-07T07:33:45.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f92af7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f92af7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f5470072c00 msgr2=0x7f5470075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f5470072c00 0x7f5470075020 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 20 client.4336.objecter put_session s=0x7f5470072530 osd=1 1 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 20 client.4336.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 10 client.4336.objecter successfully canceled tick 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f547006c820 msgr2=0x7f547006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f547006c820 0x7f547006ece0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5488000fb0 tx=0x7f5488001a60 comp rx=0 tx=0).stop 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 10 monclient: shutdown 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f548415fc80 msgr2=0x7f5484160630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 --2- 172.21.15.97:0/1218731073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f548415fc80 0x7f5484160630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f547c00afd0 tx=0x7f547c00c950 comp rx=0 tx=0).stop 2023-12-07T07:33:45.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f3190a3d640 1 --2- 172.21.15.97:0/3314475777 >> v2:172.21.15.97:3300/0 conn(0x7f318c142bf0 0x7f318c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5492109640 1 -- 172.21.15.97:0/1218731073 reap_dead start 2023-12-07T07:33:45.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 shutdown_connections 2023-12-07T07:33:45.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.724+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 >> 172.21.15.97:0/1218731073 conn(0x7f54840065d0 msgr2=0x7f54840c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.725+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 shutdown_connections 2023-12-07T07:33:45.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.725+0000 7f5493b6c640 1 -- 172.21.15.97:0/1218731073 wait complete. 2023-12-07T07:33:45.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.725+0000 7f5493b6c640 1 librados: shutdown 2023-12-07T07:33:45.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.726+0000 7f92af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.726+0000 7f92af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.726+0000 7f92af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.726+0000 7f3190a3d640 10 monclient(hunting): get_auth_request con 0x7f318c142bf0 auth_method 0 2023-12-07T07:33:45.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.726+0000 7f3190a3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.726+0000 7f3190a3d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.726+0000 7f3190a3d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.727+0000 7f3190a3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.727+0000 7f3190a3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.727+0000 7f3190a3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 1 -- 172.21.15.97:0/3952028911 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 msgr2=0x7f92b01424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 1 --2- 172.21.15.97:0/3952028911 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 0x7f92b01424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 1 -- 172.21.15.97:0/3952028911 >> v2:172.21.15.154:3300/0 conn(0x7f92b01454a0 msgr2=0x7f92b0145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 1 --2- 172.21.15.97:0/3952028911 >> v2:172.21.15.154:3300/0 conn(0x7f92b01454a0 0x7f92b0145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 1 monclient: found mon.noname-a 2023-12-07T07:33:45.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.729+0000 7f92af7fe640 1 -- 172.21.15.97:0/3952028911 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00a97f0 con 0x7f92b00a8e10 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.730+0000 7f3183fff640 1 -- 172.21.15.97:0/3314475777 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262586835 0 0) 0x7f318c0aac10 con 0x7f318c0a8e10 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.730+0000 7f3183fff640 10 monclient(hunting): _negotiate 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.730+0000 7f3183fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.730+0000 7f3183fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.730+0000 7f3183fff640 10 monclient(hunting): my global_id is 4302 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.730+0000 7f3183fff640 1 -- 172.21.15.97:0/3314475777 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3178003520 con 0x7f318c0a8e10 2023-12-07T07:33:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.732+0000 7f92af7fe640 1 --2- 172.21.15.97:0/3952028911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b00a8e10 0x7f92b0142a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009870 tx=0x7f92a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c922363025a198ad server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 1 -- 172.21.15.97:0/3314475777 >> v1:172.21.15.154:6789/0 conn(0x7f318c0a8a30 legacy=0x7f318c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 1 -- 172.21.15.97:0/3314475777 >> v1:172.21.15.97:6789/0 conn(0x7f318c0a8e10 legacy=0x7f318c13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 1 monclient: found mon.noname-a 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.733+0000 7f3190a3d640 1 -- 172.21.15.97:0/3314475777 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c0a97f0 con 0x7f318c142bf0 2023-12-07T07:33:45.743 INFO:teuthology.orchestra.run.smithi097.stdout:73014444038 2023-12-07T07:33:45.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-07T07:33:45.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.740+0000 7f3190a3d640 1 --2- 172.21.15.97:0/3314475777 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c13ecf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f317c0097a0 tx=0x7f317c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b799eaa6d8e6f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92aeffd640 1 -- 172.21.15.97:0/3952028911 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4031640 con 0x7f92b00a8e10 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92aeffd640 10 monclient: dump: 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.746 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92aeffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92aeffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92b676c640 5 monclient: authenticate success, global_id 4299 2023-12-07T07:33:45.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.741+0000 7f92b676c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:45.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.742+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5324090070 con 0x7f531c006de0 2023-12-07T07:33:45.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.742+0000 7f5319ffb640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f530806c7d0 0x7f530806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.742+0000 7f5328e56640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f530806c7d0 0x7f530806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.742+0000 7f5328e56640 10 monclient: get_auth_request con 0x7f530806c7d0 auth_method 0 2023-12-07T07:33:45.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.743+0000 7f5319ffb640 10 client.4286.objecter ms_handle_connect 0x7f530806c7d0 2023-12-07T07:33:45.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.743+0000 7f5328e56640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f530806c7d0 0x7f530806ec90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5320007fd0 tx=0x7f5320007aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92aeffd640 1 -- 172.21.15.97:0/3952028911 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a40317a0 con 0x7f92b00a8e10 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92aeffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92aeffd640 1 -- 172.21.15.97:0/3952028911 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4031b30 con 0x7f92b00a8e10 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92aeffd640 10 monclient: dump: 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.750 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 4 set_mon_vals no callback set 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 10 monclient: shutdown 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 1 -- 172.21.15.97:0/3952028911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b00a8e10 msgr2=0x7f92b0142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 1 --2- 172.21.15.97:0/3952028911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b00a8e10 0x7f92b0142a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009870 tx=0x7f92a402b8f0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 1 -- 172.21.15.97:0/3952028911 shutdown_connections 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 1 --2- 172.21.15.97:0/3952028911 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b00a8e10 0x7f92b0142a20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 1 --2- 172.21.15.97:0/3952028911 >> v2:172.21.15.154:3300/0 conn(0x7f92b01454a0 0x7f92b0145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.746+0000 7f92b676c640 1 --2- 172.21.15.97:0/3952028911 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 0x7f92b01424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f92b676c640 1 -- 172.21.15.97:0/3952028911 >> 172.21.15.97:0/3952028911 conn(0x7f92b0061a30 msgr2=0x7f92b00b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f92b676c640 1 -- 172.21.15.97:0/3952028911 shutdown_connections 2023-12-07T07:33:45.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f92b676c640 1 -- 172.21.15.97:0/3952028911 wait complete. 2023-12-07T07:33:45.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 1 -- 172.21.15.97:0/3314475777 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c030610 con 0x7f318c142bf0 2023-12-07T07:33:45.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f92b676c640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f92b676c640 10 monclient: monmap: 2023-12-07T07:33:45.752 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.752 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: dump: 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 1 -- 172.21.15.97:0/3314475777 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c030770 con 0x7f318c142bf0 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 1 -- 172.21.15.97:0/3314475777 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c030b00 con 0x7f318c142bf0 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f3183fff640 10 monclient: dump: 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.756 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.756 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f31934a2640 5 monclient: authenticate success, global_id 4305 2023-12-07T07:33:45.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f31934a2640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.747+0000 7f31934a2640 4 set_mon_vals no callback set 2023-12-07T07:33:45.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f92b676c640 1 librados: starting msgr at 2023-12-07T07:33:45.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f92b676c640 1 librados: starting objecter 2023-12-07T07:33:45.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f92b676c640 1 Processor -- start 2023-12-07T07:33:45.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f92b676c640 1 -- start start 2023-12-07T07:33:45.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f92b676c640 1 librados: setting wanted keys 2023-12-07T07:33:45.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f92b676c640 1 librados: calling monclient init 2023-12-07T07:33:45.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f31934a2640 10 monclient: shutdown 2023-12-07T07:33:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f31934a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f31934a2640 1 -- 172.21.15.97:0/3314475777 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 msgr2=0x7f318c13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f31934a2640 1 --2- 172.21.15.97:0/3314475777 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c13ecf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f317c0097a0 tx=0x7f317c02b960 comp rx=0 tx=0).stop 2023-12-07T07:33:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f92b676c640 10 monclient: init 2023-12-07T07:33:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f31934a2640 1 -- 172.21.15.97:0/3314475777 shutdown_connections 2023-12-07T07:33:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f31934a2640 1 --2- 172.21.15.97:0/3314475777 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c13ecf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.748+0000 7f31934a2640 1 -- 172.21.15.97:0/3314475777 >> 172.21.15.97:0/3314475777 conn(0x7f318c061a30 msgr2=0x7f318c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f31934a2640 1 -- 172.21.15.97:0/3314475777 shutdown_connections 2023-12-07T07:33:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f31934a2640 1 -- 172.21.15.97:0/3314475777 wait complete. 2023-12-07T07:33:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 0x7f92b00ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f31934a2640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f31934a2640 10 monclient: monmap: 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.760 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): picked mon.noname-c con 0x7f92b00a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92b00a8e10 0x7f92b00b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): picked mon.noname-b con 0x7f92b00a8e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b01454a0 0x7f92b00b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): picked mon.noname-a con 0x7f92b01454a0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00b0540 con 0x7f92b01454a0 2023-12-07T07:33:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00ae170 con 0x7f92b00a8e10 2023-12-07T07:33:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b01d4aa0 con 0x7f92b00a8a30 2023-12-07T07:33:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.749+0000 7f92b676c640 10 monclient(hunting): authenticate will time out at 1348.128188s 2023-12-07T07:33:45.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f31934a2640 1 librados: starting msgr at 2023-12-07T07:33:45.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f31934a2640 1 librados: starting objecter 2023-12-07T07:33:45.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f31934a2640 1 Processor -- start 2023-12-07T07:33:45.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f31934a2640 1 -- start start 2023-12-07T07:33:45.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f31934a2640 1 librados: setting wanted keys 2023-12-07T07:33:45.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f31934a2640 1 librados: calling monclient init 2023-12-07T07:33:45.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f31934a2640 10 monclient: init 2023-12-07T07:33:45.764 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.750+0000 7f92b4d09640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b01454a0 0x7f92b00b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.764 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b01454a0 0x7f92b00b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38944/0 (socket says 172.21.15.97:38944) 2023-12-07T07:33:45.764 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 1 -- 172.21.15.97:0/3334993634 learned_addr learned my addr 172.21.15.97:0/3334993634 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.764 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 10 monclient(hunting): get_auth_request con 0x7f92b01454a0 auth_method 0 2023-12-07T07:33:45.764 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.764 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.764 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f318c0a8a30 0x7f318c144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): picked mon.noname-c con 0x7f318c0a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318c0a8e10 0x7f318c1389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): picked mon.noname-b con 0x7f318c0a8e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c1453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): picked mon.noname-a con 0x7f318c142bf0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c139530 con 0x7f318c142bf0 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c145f60 con 0x7f318c0a8e10 2023-12-07T07:33:45.765 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318c145ad0 con 0x7f318c0a8a30 2023-12-07T07:33:45.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.766 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f31934a2640 10 monclient(hunting): authenticate will time out at 1348.129847s 2023-12-07T07:33:45.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.751+0000 7f92b4d09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2023-12-07T07:33:45.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 1 -- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 msgr2=0x7f92b00ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 0x7f92b00ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 1 -- 172.21.15.97:0/3334993634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92b00a8e10 msgr2=0x7f92b00b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92b00a8e10 0x7f92b00b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 1 monclient: found mon.noname-a 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.754+0000 7f92b4d09640 1 -- 172.21.15.97:0/3334993634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40094d0 con 0x7f92b01454a0 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.757+0000 7f92b4d09640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b01454a0 0x7f92b00b0ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b970 tx=0x7f92a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.757+0000 7f92acff9640 10 client.?.objecter ms_handle_connect 0x7f92b01454a0 2023-12-07T07:33:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.757+0000 7f92acff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f53240d2b30 con 0x7f531c006de0 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 10 client.4286.objecter ms_dispatch 0x7f5324000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 3 client.4286.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 3 client.4286.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 10 client.4286.objecter checking command 1 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7f5308072bb0 0x7f5308074fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 20 client.4286.objecter _get_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 20 client.4286.objecter put_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 15 client.4286.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 20 client.4286.objecter _get_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 20 client.4286.objecter get_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 15 client.4286.objecter _session_command_op_assign 7 1 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 20 client.4286.objecter put_session s=0x7f53080724e0 osd=7 4 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 10 client.4286.objecter _send_command 1 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 --> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5308075680 con 0x7f5308072bb0 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 20 client.4286.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53240ce770 con 0x7f531c006de0 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.758+0000 7f5319ffb640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.759+0000 7f5329e58640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7f5308072bb0 0x7f5308074fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.759+0000 7f5329e58640 10 monclient: get_auth_request con 0x7f5308072bb0 auth_method 0 2023-12-07T07:33:45.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.761+0000 7f3191a3f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c1453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.765+0000 7f5329e58640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7f5308072bb0 0x7f5308074fd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.765+0000 7f5319ffb640 10 client.4286.objecter ms_handle_connect 0x7f5308072bb0 2023-12-07T07:33:45.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.780+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd4e400ab70 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.780+0000 7fd4daffd640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd4cc06c820 0x7fd4cc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.780+0000 7fd4e9d9b640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd4cc06c820 0x7fd4cc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.780+0000 7fd4e9d9b640 10 monclient: get_auth_request con 0x7fd4cc06c820 auth_method 0 2023-12-07T07:33:45.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.781+0000 7fd4daffd640 10 client.4289.objecter ms_handle_connect 0x7fd4cc06c820 2023-12-07T07:33:45.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.781+0000 7fd4e9d9b640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd4cc06c820 0x7fd4cc06ece0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec072780 tx=0x7fd4dc007930 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd4e400ae20 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 10 client.4289.objecter ms_dispatch 0x7fd4ec000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 3 client.4289.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 3 client.4289.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 10 client.4289.objecter checking command 1 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fd4cc072c00 0x7fd4cc075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 20 client.4289.objecter _get_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 20 client.4289.objecter put_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 15 client.4289.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 20 client.4289.objecter _get_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 20 client.4289.objecter get_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 15 client.4289.objecter _session_command_op_assign 4 1 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 20 client.4289.objecter put_session s=0x7fd4cc072530 osd=4 4 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 10 client.4289.objecter _send_command 1 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 --> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc0756d0 con 0x7fd4cc072c00 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 20 client.4289.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4e4054fd0 con 0x7fd4ec06f2b0 2023-12-07T07:33:45.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.783+0000 7fd4daffd640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:33:45.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.784+0000 7fd4ead9d640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fd4cc072c00 0x7fd4cc075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.784+0000 7fd4ead9d640 10 monclient: get_auth_request con 0x7fd4cc072c00 auth_method 0 2023-12-07T07:33:45.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38005ce0 con 0x7fdb4413cf10 2023-12-07T07:33:45.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: dump: 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb38005e40 con 0x7fdb4413cf10 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb380072b0 con 0x7fdb4413cf10 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb32ffd640 10 monclient: dump: 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.790 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb40d99640 4 set_mon_vals no callback set 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb48ad0640 5 monclient: authenticate success, global_id 4292 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb48ad0640 10 monclient: _renew_subs 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb48ad0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb44135d60 con 0x7fdb4413cf10 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.786+0000 7fdb48ad0640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.787+0000 7fdb48ad0640 10 monclient: _renew_subs 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.787+0000 7fdb48ad0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.787+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb44142fd0 con 0x7fdb4413cf10 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.787+0000 7fdb48ad0640 1 librados: init done 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.788+0000 7fdb48ad0640 10 client.4292.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.788+0000 7fdb48ad0640 15 client.4292.objecter _session_command_op_assign -1 1 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.788+0000 7fdb48ad0640 20 client.4292.objecter _get_session osd=-1 returning homeless 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.788+0000 7fdb48ad0640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.788+0000 7fdb48ad0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.788+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb440b1870 con 0x7fdb4413cf10 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.789+0000 7fd4daffd640 10 client.4289.objecter ms_handle_connect 0x7fd4cc072c00 2023-12-07T07:33:45.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.789+0000 7fd4ead9d640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fd4cc072c00 0x7fd4cc075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.801+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffaf40bac80 con 0x7ffaec0355b0 2023-12-07T07:33:45.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.801+0000 7ffaeaffd640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7ffad806c7d0 0x7ffad806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.804 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.802+0000 7ffaf9b90640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7ffad806c7d0 0x7ffad806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.804 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.802+0000 7ffaf9b90640 10 monclient: get_auth_request con 0x7ffad806c7d0 auth_method 0 2023-12-07T07:33:45.804 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.802+0000 7ffaeaffd640 10 client.4283.objecter ms_handle_connect 0x7ffad806c7d0 2023-12-07T07:33:45.804 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.802+0000 7ffaf9b90640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7ffad806c7d0 0x7ffad806ec90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0c34a0 tx=0x7ffae4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.807 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7f3191a3f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c1453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38950/0 (socket says 172.21.15.97:38950) 2023-12-07T07:33:45.807 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7f3191a3f640 1 -- 172.21.15.97:0/3933470383 learned_addr learned my addr 172.21.15.97:0/3933470383 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.807 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7f3191a3f640 10 monclient(hunting): get_auth_request con 0x7f318c142bf0 auth_method 0 2023-12-07T07:33:45.807 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7f3191a3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.807 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7f3191a3f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.807 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7f3191a3f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.807 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffaf40a6e40 con 0x7ffaec0355b0 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 10 client.4283.objecter ms_dispatch 0x7ffaf4000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 3 client.4283.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 3 client.4283.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 10 client.4283.objecter checking command 1 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7ffad8072bb0 0x7ffad8074fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 20 client.4283.objecter _get_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 20 client.4283.objecter put_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 15 client.4283.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 20 client.4283.objecter _get_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 20 client.4283.objecter get_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 15 client.4283.objecter _session_command_op_assign 3 1 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 20 client.4283.objecter put_session s=0x7ffad80724e0 osd=3 4 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 10 client.4283.objecter _send_command 1 2023-12-07T07:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 --> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffad8075680 con 0x7ffad8072bb0 2023-12-07T07:33:45.809 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 20 client.4283.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.809 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffaf4067300 con 0x7ffaec0355b0 2023-12-07T07:33:45.809 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.805+0000 7ffaeaffd640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:33:45.809 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.806+0000 7ffafab92640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7ffad8072bb0 0x7ffad8074fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.809 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.806+0000 7ffafab92640 10 monclient: get_auth_request con 0x7ffad8072bb0 auth_method 0 2023-12-07T07:33:45.810 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.808+0000 7f3191a3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.810 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.808+0000 7f3191a3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.810 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.808+0000 7f3191a3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.812 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.810+0000 7ffaeaffd640 10 client.4283.objecter ms_handle_connect 0x7ffad8072bb0 2023-12-07T07:33:45.813 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.810+0000 7ffafab92640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7ffad8072bb0 0x7ffad8074fd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.817+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== osd.7 v2:172.21.15.154:6816/1161771449 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5308075680 con 0x7f5308072bb0 2023-12-07T07:33:45.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.817+0000 7f5319ffb640 10 client.4286.objecter ms_dispatch 0x7f5324000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.820 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.817+0000 7f5319ffb640 10 client.4286.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.820 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.817+0000 7f5319ffb640 20 client.4286.objecter put_session s=0x7f53080724e0 osd=7 4 2023-12-07T07:33:45.820 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.817+0000 7f5319ffb640 15 client.4286.objecter _session_command_op_remove 7 1 2023-12-07T07:33:45.820 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 1 -- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f318c0a8a30 msgr2=0x7f318c144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f318c0a8a30 0x7f318c144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 1 -- 172.21.15.97:0/3933470383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318c0a8e10 msgr2=0x7f318c1389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318c0a8e10 0x7f318c1389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 1 monclient: found mon.noname-a 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.821 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.818+0000 7f3191a3f640 1 -- 172.21.15.97:0/3933470383 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c009400 con 0x7f318c142bf0 2023-12-07T07:33:45.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 10 client.4286.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 15 client.4286.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 20 client.4286.objecter _get_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 20 client.4286.objecter put_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 20 client.4286.objecter _get_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 15 client.4286.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 20 client.4286.objecter get_session s=0x7f53080724e0 osd=7 3 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 15 client.4286.objecter _session_command_op_assign 7 2 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 20 client.4286.objecter put_session s=0x7f53080724e0 osd=7 4 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 10 client.4286.objecter _send_command 2 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.835+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 --> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f531c02f060 con 0x7f5308072bb0 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.836+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== osd.4 v2:172.21.15.154:6808/1781054720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd4cc0756d0 con 0x7fd4cc072c00 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.836+0000 7fd4daffd640 10 client.4289.objecter ms_dispatch 0x7fd4ec000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.836+0000 7fd4daffd640 10 client.4289.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.836+0000 7fd4daffd640 20 client.4289.objecter put_session s=0x7fd4cc072530 osd=4 4 2023-12-07T07:33:45.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.836+0000 7fd4daffd640 15 client.4289.objecter _session_command_op_remove 4 1 2023-12-07T07:33:45.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.840+0000 7f5319ffb640 1 -- 172.21.15.97:0/1917151265 <== osd.7 v2:172.21.15.154:6816/1161771449 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f531c02f060 con 0x7f5308072bb0 2023-12-07T07:33:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.840+0000 7f5319ffb640 10 client.4286.objecter ms_dispatch 0x7f5324000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.840+0000 7f5319ffb640 10 client.4286.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.840+0000 7f5319ffb640 20 client.4286.objecter put_session s=0x7f53080724e0 osd=7 4 2023-12-07T07:33:45.843 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.840+0000 7f5319ffb640 15 client.4286.objecter _session_command_op_remove 7 2 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 10 librados: watch_flush enter 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 10 librados: watch_flush exit 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 10 client.4286.objecter close_session for osd.7 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7f5308072bb0 msgr2=0x7f5308074fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7f5308072bb0 0x7f5308074fd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 20 client.4286.objecter put_session s=0x7f53080724e0 osd=7 1 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 20 client.4286.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 10 client.4286.objecter successfully canceled tick 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f530806c7d0 msgr2=0x7f530806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f530806c7d0 0x7f530806ec90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5320007fd0 tx=0x7f5320007aa0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 10 monclient: shutdown 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c006de0 msgr2=0x7f531c02f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 --2- 172.21.15.97:0/1917151265 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f531c006de0 0x7f531c02f3a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5324000b60 tx=0x7f5324062060 comp rx=0 tx=0).stop 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f5329e58640 1 -- 172.21.15.97:0/1917151265 reap_dead start 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 shutdown_connections 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.841+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 >> 172.21.15.97:0/1917151265 conn(0x7f531c005ea0 msgr2=0x7f531c006280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.842+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 shutdown_connections 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.842+0000 7f532a659640 1 -- 172.21.15.97:0/1917151265 wait complete. 2023-12-07T07:33:45.846 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.843+0000 7f532a659640 1 librados: shutdown 2023-12-07T07:33:45.847 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.845+0000 7f3191a3f640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c1453e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f318400cb70 tx=0x7f318400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.847 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.845+0000 7f3181ffb640 10 client.?.objecter ms_handle_connect 0x7f318c142bf0 2023-12-07T07:33:45.847 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.845+0000 7f3181ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.851 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 1 -- 172.21.15.97:0/3334993634 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800cb40 con 0x7f92b01454a0 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: dump: 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.852 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 1 -- 172.21.15.97:0/3334993634 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a800ccc0 con 0x7f92b01454a0 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 5 monclient: authenticate success, global_id 4308 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 10 monclient: _renew_subs 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 1 -- 172.21.15.97:0/3334993634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b013b700 con 0x7f92b01454a0 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.853 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 10 monclient: _renew_subs 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 1 -- 172.21.15.97:0/3334993634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b01d6e60 con 0x7f92b01454a0 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92b676c640 1 librados: init done 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 1 -- 172.21.15.97:0/3334993634 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8014680 con 0x7f92b01454a0 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92ae7fc640 4 set_mon_vals no callback set 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.846+0000 7f92acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.848+0000 7f92acff9640 10 monclient: dump: 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.854 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.848+0000 7f92967fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.847+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdb38007410 con 0x7fdb4413cf10 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.847+0000 7fdb32ffd640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fdb2406c820 0x7fdb2406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.849+0000 7fdb4259c640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fdb2406c820 0x7fdb2406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.849+0000 7fdb4259c640 10 monclient: get_auth_request con 0x7fdb2406c820 auth_method 0 2023-12-07T07:33:45.855 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.850+0000 7fdb4259c640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fdb2406c820 0x7fdb2406ece0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0043c0 tx=0x7fdb3c007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.850+0000 7fdb32ffd640 10 client.4292.objecter ms_handle_connect 0x7fdb2406c820 2023-12-07T07:33:45.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.849+0000 7f92967fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.849+0000 7f92967fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.849+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b0067ac0 con 0x7f92b01454a0 2023-12-07T07:33:45.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdb38058890 con 0x7fdb4413cf10 2023-12-07T07:33:45.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 10 client.4292.objecter ms_dispatch 0x7fdb44000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 3 client.4292.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 3 client.4292.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 10 client.4292.objecter checking command 1 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7fdb24073670 0x7fdb24075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 20 client.4292.objecter _get_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 20 client.4292.objecter put_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 15 client.4292.objecter _session_command_op_remove -1 1 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 20 client.4292.objecter _get_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 20 client.4292.objecter get_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 15 client.4292.objecter _session_command_op_assign 6 1 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 20 client.4292.objecter put_session s=0x7fdb24072fa0 osd=6 4 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 10 client.4292.objecter _send_command 1 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 --> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb2407a5f0 con 0x7fdb24073670 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 20 client.4292.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb38057f60 con 0x7fdb4413cf10 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.852+0000 7fdb32ffd640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 1 -- 172.21.15.97:0/3933470383 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318400ad40 con 0x7f318c142bf0 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: dump: 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.858 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.859 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.859 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.859 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.859 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.859 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 1 -- 172.21.15.97:0/3933470383 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184012400 con 0x7f318c142bf0 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 1 -- 172.21.15.97:0/3933470383 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318400aea0 con 0x7f318c142bf0 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f3181ffb640 10 monclient: dump: 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.860 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.861 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.861 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.861 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.861 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.861 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7f31837fe640 4 set_mon_vals no callback set 2023-12-07T07:33:45.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7fdb42d9d640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7fdb24073670 0x7fdb24075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.853+0000 7fdb42d9d640 10 monclient: get_auth_request con 0x7fdb24073670 auth_method 0 2023-12-07T07:33:45.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 5 monclient: authenticate success, global_id 4311 2023-12-07T07:33:45.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 10 monclient: _renew_subs 2023-12-07T07:33:45.862 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318c1d7030 con 0x7f318c142bf0 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 10 monclient: _renew_subs 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c1d7460 con 0x7f318c142bf0 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.854+0000 7f31934a2640 1 librados: init done 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 10 client.4289.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 15 client.4289.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 20 client.4289.objecter _get_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 20 client.4289.objecter put_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 20 client.4289.objecter _get_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.863 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 15 client.4289.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 20 client.4289.objecter get_session s=0x7fd4cc072530 osd=4 3 2023-12-07T07:33:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 15 client.4289.objecter _session_command_op_assign 4 2 2023-12-07T07:33:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 20 client.4289.objecter put_session s=0x7fd4cc072530 osd=4 4 2023-12-07T07:33:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 10 client.4289.objecter _send_command 2 2023-12-07T07:33:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.858+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 --> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd4d00011c0 con 0x7fd4cc072c00 2023-12-07T07:33:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.857+0000 7fdb42d9d640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7fdb24073670 0x7fdb24075b10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.864 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.857+0000 7fdb32ffd640 10 client.4292.objecter ms_handle_connect 0x7fdb24073670 2023-12-07T07:33:45.865 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.861+0000 7f31934a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.865 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.861+0000 7f31934a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.865 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.861+0000 7f31934a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:45.865 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.861+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318c067ac0 con 0x7f318c142bf0 2023-12-07T07:33:45.867 INFO:teuthology.orchestra.run.smithi097.stdout:55834574854 2023-12-07T07:33:45.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-07T07:33:45.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.865+0000 7fd4daffd640 1 -- 172.21.15.97:0/3303074175 <== osd.4 v2:172.21.15.154:6808/1781054720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd4d00011c0 con 0x7fd4cc072c00 2023-12-07T07:33:45.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.865+0000 7fd4daffd640 10 client.4289.objecter ms_dispatch 0x7fd4ec000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.865+0000 7fd4daffd640 10 client.4289.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.865+0000 7fd4daffd640 20 client.4289.objecter put_session s=0x7fd4cc072530 osd=4 4 2023-12-07T07:33:45.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.865+0000 7fd4daffd640 15 client.4289.objecter _session_command_op_remove 4 2 2023-12-07T07:33:45.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 10 librados: watch_flush enter 2023-12-07T07:33:45.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 10 librados: watch_flush exit 2023-12-07T07:33:45.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 10 client.4289.objecter close_session for osd.4 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fd4cc072c00 msgr2=0x7fd4cc075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fd4cc072c00 0x7fd4cc075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 20 client.4289.objecter put_session s=0x7fd4cc072530 osd=4 1 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 20 client.4289.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 10 client.4289.objecter successfully canceled tick 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd4cc06c820 msgr2=0x7fd4cc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd4cc06c820 0x7fd4cc06ece0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec072780 tx=0x7fd4dc007930 comp rx=0 tx=0).stop 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 10 monclient: shutdown 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec06f2b0 msgr2=0x7fd4ec0a9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 --2- 172.21.15.97:0/3303074175 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ec06f2b0 0x7fd4ec0a9f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400c850 tx=0x7fd4e400cd10 comp rx=0 tx=0).stop 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4ead9d640 1 -- 172.21.15.97:0/3303074175 reap_dead start 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 shutdown_connections 2023-12-07T07:33:45.870 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 >> 172.21.15.97:0/3303074175 conn(0x7fd4ec0b01e0 msgr2=0x7fd4ec149c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 shutdown_connections 2023-12-07T07:33:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 -- 172.21.15.97:0/3303074175 wait complete. 2023-12-07T07:33:45.871 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.866+0000 7fd4f0fb0640 1 librados: shutdown 2023-12-07T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.873+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== osd.3 v2:172.21.15.97:6810/929626392 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ffad8075680 con 0x7ffad8072bb0 2023-12-07T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.873+0000 7ffaeaffd640 10 client.4283.objecter ms_dispatch 0x7ffaf4000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.873+0000 7ffaeaffd640 10 client.4283.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.873+0000 7ffaeaffd640 20 client.4283.objecter put_session s=0x7ffad80724e0 osd=3 4 2023-12-07T07:33:45.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.873+0000 7ffaeaffd640 15 client.4283.objecter _session_command_op_remove 3 1 2023-12-07T07:33:45.883 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.881+0000 7f70f5783640 10 monclient: get_monmap_and_config 2023-12-07T07:33:45.883 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.881+0000 7f70f5783640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.883 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.881+0000 7f70f5783640 10 monclient: monmap: 2023-12-07T07:33:45.883 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.883 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.883 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:45.882539+0000 2023-12-07T07:33:45.883 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:45.882539+0000 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:45.884 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.885 INFO:teuthology.orchestra.run.smithi097.stdout:55834574853 2023-12-07T07:33:45.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.882+0000 7f70f5783640 1 Processor -- start 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.882+0000 7f70f5783640 1 -- start start 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.882+0000 7f70f5783640 10 monclient: init 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70f006f0f0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f006f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient(hunting): picked mon.noname-b con 0x7f70f00ae350 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f70f0072330 0x7f70f0141b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient(hunting): picked mon.noname-a con 0x7f70f0072330 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:45.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f0071340 con 0x7f70f006f0f0 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f0071620 con 0x7f70f0072330 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00702f0 con 0x7f70f00ae350 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.883+0000 7f70f5783640 10 monclient(hunting): authenticate will time out at 1348.261885s 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.884+0000 7f70ee7fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f006f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.884+0000 7f70ee7fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f006f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38168/0 (socket says 172.21.15.97:38168) 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.884+0000 7f70ee7fc640 1 -- 172.21.15.97:0/2684510473 learned_addr learned my addr 172.21.15.97:0/2684510473 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.884+0000 7f70ee7fc640 10 monclient(hunting): get_auth_request con 0x7f70f00ae350 auth_method 0 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.884+0000 7f70ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.884+0000 7f70ee7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.884+0000 7f70ee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.885+0000 7f70ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.885+0000 7f70ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.889 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.885+0000 7f70ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.890 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 1 -- 172.21.15.97:0/2684510473 >> v2:172.21.15.97:3300/0 conn(0x7f70f0072330 msgr2=0x7f70f0141b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 1 --2- 172.21.15.97:0/2684510473 >> v2:172.21.15.97:3300/0 conn(0x7f70f0072330 0x7f70f0141b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 1 -- 172.21.15.97:0/2684510473 >> v1:172.21.15.97:6789/0 conn(0x7f70f006f0f0 legacy=0x7f70f0071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 1 monclient: found mon.noname-b 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.891 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.888+0000 7f70ee7fc640 1 -- 172.21.15.97:0/2684510473 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0141740 con 0x7f70f00ae350 2023-12-07T07:33:45.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.891+0000 7f70ee7fc640 1 --2- 172.21.15.97:0/2684510473 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f006f4d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009870 tx=0x7f70e402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80f9a9ca0a0fc23b server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.891+0000 7f92acff9640 1 -- 172.21.15.97:0/3334993634 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f92a801d3b0 con 0x7f92b01454a0 2023-12-07T07:33:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.892+0000 7f92acff9640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f929c06c820 0x7f929c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.893+0000 7f92affff640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f929c06c820 0x7f929c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.893+0000 7f92affff640 10 monclient: get_auth_request con 0x7f929c06c820 auth_method 0 2023-12-07T07:33:45.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.893+0000 7f92acff9640 10 client.4308.objecter ms_handle_connect 0x7f929c06c820 2023-12-07T07:33:45.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.893+0000 7f92affff640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f929c06c820 0x7f929c06ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009710 tx=0x7f92a0006c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.894+0000 7f70edffb640 1 -- 172.21.15.97:0/2684510473 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e40317c0 con 0x7f70f00ae350 2023-12-07T07:33:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.894+0000 7f70edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.894+0000 7f70edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.894+0000 7f70edffb640 10 monclient: dump: 2023-12-07T07:33:45.897 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.894+0000 7f70edffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.894+0000 7f70edffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.894+0000 7f70edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.895+0000 7f70f5783640 5 monclient: authenticate success, global_id 4298 2023-12-07T07:33:45.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.895+0000 7f70f5783640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:45.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70edffb640 1 -- 172.21.15.97:0/2684510473 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e4031920 con 0x7f70f00ae350 2023-12-07T07:33:45.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70edffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70edffb640 1 -- 172.21.15.97:0/2684510473 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e40317c0 con 0x7f70f00ae350 2023-12-07T07:33:45.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70edffb640 10 monclient: dump: 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70f5783640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.896+0000 7f70f5783640 4 set_mon_vals no callback set 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 10 monclient: shutdown 2023-12-07T07:33:45.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 1 -- 172.21.15.97:0/2684510473 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 msgr2=0x7f70f006f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 1 --2- 172.21.15.97:0/2684510473 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f006f4d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009870 tx=0x7f70e402b860 comp rx=0 tx=0).stop 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 1 -- 172.21.15.97:0/2684510473 shutdown_connections 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 1 --2- 172.21.15.97:0/2684510473 >> v2:172.21.15.97:3300/0 conn(0x7f70f0072330 0x7f70f0141b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 1 --2- 172.21.15.97:0/2684510473 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f006f4d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.897+0000 7f70f5783640 1 -- 172.21.15.97:0/2684510473 >> 172.21.15.97:0/2684510473 conn(0x7f70f00b3e30 msgr2=0x7f70f00b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7f70f5783640 1 -- 172.21.15.97:0/2684510473 shutdown_connections 2023-12-07T07:33:45.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7f70f5783640 1 -- 172.21.15.97:0/2684510473 wait complete. 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 10 client.4283.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 15 client.4283.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 20 client.4283.objecter _get_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 20 client.4283.objecter put_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 20 client.4283.objecter _get_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 15 client.4283.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 20 client.4283.objecter get_session s=0x7ffad80724e0 osd=3 3 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 15 client.4283.objecter _session_command_op_assign 3 2 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 20 client.4283.objecter put_session s=0x7ffad80724e0 osd=3 4 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 10 client.4283.objecter _send_command 2 2023-12-07T07:33:45.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.898+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 --> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffaec0c9ae0 con 0x7ffad8072bb0 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.899+0000 7f70f5783640 10 monclient: build_initial_monmap 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.899+0000 7f70f5783640 10 monclient: monmap: 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:45.906 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.899+0000 7f70f5783640 1 librados: starting msgr at 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.899+0000 7f70f5783640 1 librados: starting objecter 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.900+0000 7f70f5783640 1 Processor -- start 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.900+0000 7f70f5783640 1 -- start start 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.900+0000 7f70f5783640 1 librados: setting wanted keys 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.900+0000 7f70f5783640 1 librados: calling monclient init 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.900+0000 7f70f5783640 10 monclient: init 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.901+0000 7f70f5783640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.901+0000 7f70f5783640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.901+0000 7f70f5783640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f006f0f0 0x7f70f011ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.901+0000 7f70f5783640 10 monclient(hunting): picked mon.noname-b con 0x7f70f006f0f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.901+0000 7f70f5783640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70f0072330 0x7f70f011fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.901+0000 7f70f5783640 10 monclient(hunting): picked mon.noname-a con 0x7f70f0072330 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.901+0000 7f70f5783640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f01d7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 10 monclient(hunting): picked mon.noname-c con 0x7f70f00ae350 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01d7390 con 0x7f70f0072330 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01201a0 con 0x7f70f006f0f0 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0121320 con 0x7f70f00ae350 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:45.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.902+0000 7f70f5783640 10 monclient(hunting): authenticate will time out at 1348.280872s 2023-12-07T07:33:45.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.904+0000 7f70ef7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f01d7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.905+0000 7f70ef7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f01d7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38172/0 (socket says 172.21.15.97:38172) 2023-12-07T07:33:45.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.905+0000 7f70ef7fe640 1 -- 172.21.15.97:0/3811515102 learned_addr learned my addr 172.21.15.97:0/3811515102 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:45.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.905+0000 7f70ef7fe640 10 monclient(hunting): get_auth_request con 0x7f70f00ae350 auth_method 0 2023-12-07T07:33:45.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.905+0000 7f70ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.905+0000 7f70ef7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.905+0000 7f70ef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.906+0000 7f70ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.906+0000 7f70ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.906+0000 7f70ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.907+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== osd.6 v2:172.21.15.154:6824/1615374996 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fdb2407a5f0 con 0x7fdb24073670 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.907+0000 7fdb32ffd640 10 client.4292.objecter ms_dispatch 0x7fdb44000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.907+0000 7fdb32ffd640 10 client.4292.objecter _finish_command 1 = system:0 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.907+0000 7fdb32ffd640 20 client.4292.objecter put_session s=0x7fdb24072fa0 osd=6 4 2023-12-07T07:33:45.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.907+0000 7fdb32ffd640 15 client.4292.objecter _session_command_op_remove 6 1 2023-12-07T07:33:45.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.908+0000 7ffaeaffd640 1 -- 172.21.15.97:0/3904395677 <== osd.3 v2:172.21.15.97:6810/929626392 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffaf000a2f0 con 0x7ffad8072bb0 2023-12-07T07:33:45.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.908+0000 7ffaeaffd640 10 client.4283.objecter ms_dispatch 0x7ffaf4000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.908+0000 7ffaeaffd640 10 client.4283.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.908+0000 7ffaeaffd640 20 client.4283.objecter put_session s=0x7ffad80724e0 osd=3 4 2023-12-07T07:33:45.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.908+0000 7ffaeaffd640 15 client.4283.objecter _session_command_op_remove 3 2 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.908+0000 7ffafb393640 10 librados: watch_flush enter 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 10 librados: watch_flush exit 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 10 client.4283.objecter close_session for osd.3 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7ffad8072bb0 msgr2=0x7ffad8074fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7ffad8072bb0 0x7ffad8074fd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 20 client.4283.objecter put_session s=0x7ffad80724e0 osd=3 1 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 20 client.4283.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 10 client.4283.objecter successfully canceled tick 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7ffad806c7d0 msgr2=0x7ffad806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7ffad806c7d0 0x7ffad806ec90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0c34a0 tx=0x7ffae4009190 comp rx=0 tx=0).stop 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 10 monclient: shutdown 2023-12-07T07:33:45.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 msgr2=0x7ffaec033220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafb393640 1 --2- 172.21.15.97:0/3904395677 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ffaec0355b0 0x7ffaec033220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4074c70 tx=0x7ffaf40691a0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.909+0000 7ffafab92640 1 -- 172.21.15.97:0/3904395677 reap_dead start 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.910+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 shutdown_connections 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.910+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 >> 172.21.15.97:0/3904395677 conn(0x7ffaec006390 msgr2=0x7ffaec0c87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.910+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 shutdown_connections 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.910+0000 7ffafb393640 1 -- 172.21.15.97:0/3904395677 wait complete. 2023-12-07T07:33:45.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.910+0000 7ffafb393640 1 librados: shutdown 2023-12-07T07:33:45.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2023-12-07T07:33:45.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 1 -- 172.21.15.97:0/3811515102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f006f0f0 msgr2=0x7f70f011ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f006f0f0 0x7f70f011ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 1 -- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70f0072330 msgr2=0x7f70f011fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70f0072330 0x7f70f011fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 1 monclient: found mon.noname-c 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.920+0000 7f70ef7fe640 1 -- 172.21.15.97:0/3811515102 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e40094d0 con 0x7f70f00ae350 2023-12-07T07:33:45.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.923+0000 7f70d77fe640 10 client.?.objecter ms_handle_connect 0x7f70f00ae350 2023-12-07T07:33:45.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.923+0000 7f70d77fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:45.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.923+0000 7f70ef7fe640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f01d7820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b970 tx=0x7f70e800be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.926 INFO:teuthology.orchestra.run.smithi097.stdout:73014444037 2023-12-07T07:33:45.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-07T07:33:45.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.927+0000 7f70d77fe640 1 -- 172.21.15.97:0/3811515102 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800cb40 con 0x7f70f00ae350 2023-12-07T07:33:45.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:45.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: dump: 2023-12-07T07:33:45.930 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.930 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 1 -- 172.21.15.97:0/3811515102 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800ccc0 con 0x7f70f00ae350 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 1 -- 172.21.15.97:0/3811515102 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8014680 con 0x7f70f00ae350 2023-12-07T07:33:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70d77fe640 10 monclient: dump: 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70ed7fa640 4 set_mon_vals no callback set 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 5 monclient: authenticate success, global_id 4301 2023-12-07T07:33:45.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 10 monclient: _renew_subs 2023-12-07T07:33:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 1 -- 172.21.15.97:0/3811515102 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f011e010 con 0x7f70f00ae350 2023-12-07T07:33:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 10 monclient: _renew_subs 2023-12-07T07:33:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 1 -- 172.21.15.97:0/3811515102 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f011c4b0 con 0x7f70f00ae350 2023-12-07T07:33:45.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.928+0000 7f70f5783640 1 librados: init done 2023-12-07T07:33:45.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.931+0000 7f70d57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.931+0000 7f70d57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.931+0000 7f70d57fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:45.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.931+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f0075b60 con 0x7f70f00ae350 2023-12-07T07:33:45.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 10 client.4292.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 15 client.4292.objecter _session_command_op_assign -1 2 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 20 client.4292.objecter _get_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 20 client.4292.objecter put_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 20 client.4292.objecter _get_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 15 client.4292.objecter _session_command_op_remove -1 2 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 20 client.4292.objecter get_session s=0x7fdb24072fa0 osd=6 3 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 15 client.4292.objecter _session_command_op_assign 6 2 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 20 client.4292.objecter put_session s=0x7fdb24072fa0 osd=6 4 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 10 client.4292.objecter _send_command 2 2023-12-07T07:33:45.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.935+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 --> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdb44067ac0 con 0x7fdb24073670 2023-12-07T07:33:45.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb32ffd640 1 -- 172.21.15.97:0/1694726225 <== osd.6 v2:172.21.15.154:6824/1615374996 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdb44067ac0 con 0x7fdb24073670 2023-12-07T07:33:45.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb32ffd640 10 client.4292.objecter ms_dispatch 0x7fdb44000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:33:45.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb32ffd640 10 client.4292.objecter _finish_command 2 = system:0 2023-12-07T07:33:45.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb32ffd640 20 client.4292.objecter put_session s=0x7fdb24072fa0 osd=6 4 2023-12-07T07:33:45.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb32ffd640 15 client.4292.objecter _session_command_op_remove 6 2 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 10 librados: watch_flush enter 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 10 librados: watch_flush exit 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 10 client.4292.objecter close_session for osd.6 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7fdb24073670 msgr2=0x7fdb24075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7fdb24073670 0x7fdb24075b10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 20 client.4292.objecter put_session s=0x7fdb24072fa0 osd=6 1 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 20 client.4292.objecter shutdown clearing up homeless session... 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.947+0000 7fdb48ad0640 10 client.4292.objecter successfully canceled tick 2023-12-07T07:33:45.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fdb2406c820 msgr2=0x7fdb2406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fdb2406c820 0x7fdb2406ece0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0043c0 tx=0x7fdb3c007d30 comp rx=0 tx=0).stop 2023-12-07T07:33:45.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 10 monclient: shutdown 2023-12-07T07:33:45.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:45.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413cf10 msgr2=0x7fdb44137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:45.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 --2- 172.21.15.97:0/1694726225 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb4413cf10 0x7fdb44137370 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c530 tx=0x7fdb3800c9f0 comp rx=0 tx=0).stop 2023-12-07T07:33:45.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7feb757fa640 1 -- 172.21.15.97:0/2422480110 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feb70058a20 con 0x7feb781454a0 2023-12-07T07:33:45.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7feb757fa640 10 client.4342.objecter ms_dispatch 0x7feb78000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7feb757fa640 3 client.4342.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7feb757fa640 3 client.4342.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7feb757fa640 20 client.4342.objecter dump_active .. 0 homeless 2023-12-07T07:33:45.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb42d9d640 1 -- 172.21.15.97:0/1694726225 reap_dead start 2023-12-07T07:33:45.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 shutdown_connections 2023-12-07T07:33:45.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 >> 172.21.15.97:0/1694726225 conn(0x7fdb440b2310 msgr2=0x7fdb440b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:45.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 shutdown_connections 2023-12-07T07:33:45.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 -- 172.21.15.97:0/1694726225 wait complete. 2023-12-07T07:33:45.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.948+0000 7fdb48ad0640 1 librados: shutdown 2023-12-07T07:33:45.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.956+0000 7f70d77fe640 1 -- 172.21.15.97:0/3811515102 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f70e801d3b0 con 0x7f70f00ae350 2023-12-07T07:33:45.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.956+0000 7f70d77fe640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f70dc06c7d0 0x7f70dc06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:45.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.957+0000 7f70eeffd640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f70dc06c7d0 0x7f70dc06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:45.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.957+0000 7f70eeffd640 10 monclient: get_auth_request con 0x7f70dc06c7d0 auth_method 0 2023-12-07T07:33:45.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.957+0000 7feb757fa640 1 -- 172.21.15.97:0/2422480110 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb70054d60 con 0x7feb781454a0 2023-12-07T07:33:45.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.958+0000 7feb757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:45.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.958+0000 7feb757fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:45.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.961+0000 7f70eeffd640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f70dc06c7d0 0x7f70dc06ec90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f70e00015b0 tx=0x7f70e0007990 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:45.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.961+0000 7f70d77fe640 10 client.4301.objecter ms_handle_connect 0x7f70dc06c7d0 2023-12-07T07:33:45.967 INFO:teuthology.orchestra.run.smithi097.stdout:60129542149 2023-12-07T07:33:45.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-07T07:33:45.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.965+0000 7f70d77fe640 1 -- 172.21.15.97:0/3811515102 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f70e800ce30 con 0x7f70f00ae350 2023-12-07T07:33:45.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.965+0000 7f70d77fe640 10 client.4301.objecter ms_dispatch 0x7f70f0000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:45.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.965+0000 7f70d77fe640 3 client.4301.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:45.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.965+0000 7f70d77fe640 3 client.4301.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:45.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:45.965+0000 7f70d77fe640 20 client.4301.objecter dump_active .. 0 homeless 2023-12-07T07:33:46.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.009+0000 7f92acff9640 1 -- 172.21.15.97:0/3334993634 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f92a800ce30 con 0x7f92b01454a0 2023-12-07T07:33:46.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.009+0000 7f92acff9640 10 client.4308.objecter ms_dispatch 0x7f92b0000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:46.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.009+0000 7f92acff9640 3 client.4308.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:46.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.009+0000 7f92acff9640 3 client.4308.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:46.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.009+0000 7f92acff9640 20 client.4308.objecter dump_active .. 0 homeless 2023-12-07T07:33:46.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.032+0000 7fa69bce5640 10 monclient: get_monmap_and_config 2023-12-07T07:33:46.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.032+0000 7fa69bce5640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.033+0000 7fa69bce5640 10 monclient: monmap: 2023-12-07T07:33:46.035 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.035 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.035 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:46.034077+0000 2023-12-07T07:33:46.035 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:46.034077+0000 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:46.036 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.033+0000 7fa69bce5640 1 Processor -- start 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.034+0000 7fa69bce5640 1 -- start start 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.034+0000 7fa69bce5640 10 monclient: init 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6940a8a30 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa694142bf0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa6940a8e10 0x7fa69413f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient(hunting): picked mon.noname-b con 0x7fa6940a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6940aac10 con 0x7fa694142bf0 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6940aaef0 con 0x7fa6940a8a30 2023-12-07T07:33:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6940aaa10 con 0x7fa6940a8e10 2023-12-07T07:33:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa69bce5640 10 monclient(hunting): authenticate will time out at 1348.413444s 2023-12-07T07:33:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa699a81640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa6940a8a30 0x7fa6941424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44734/0 (socket says 172.21.15.97:44734) 2023-12-07T07:33:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.035+0000 7fa699a81640 1 -- 172.21.15.97:0/425118689 learned_addr learned my addr 172.21.15.97:0/425118689 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:46.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.040+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041038666 0 0) 0x7fa6940aaef0 con 0x7fa6940a8a30 2023-12-07T07:33:46.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.040+0000 7fa698a7f640 10 monclient(hunting): _negotiate 2023-12-07T07:33:46.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.040+0000 7fa698a7f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.040+0000 7fa698a7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.042 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.040+0000 7fa698a7f640 10 monclient(hunting): my global_id is 4345 2023-12-07T07:33:46.043 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.040+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa680003520 con 0x7fa6940a8a30 2023-12-07T07:33:46.044 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.042+0000 7fa69a282640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa6940a8e10 0x7fa69413f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa69a282640 10 monclient(hunting): get_auth_request con 0x7fa6940a8e10 auth_method 0 2023-12-07T07:33:46.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa69a282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa69a282640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.045 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa69a282640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2829789642 0 0) 0x7fa680003520 con 0x7fa6940a8a30 2023-12-07T07:33:46.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa6940a8e10 msgr2=0x7fa69413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa6940a8e10 0x7fa69413f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 >> v1:172.21.15.97:6789/0 conn(0x7fa694142bf0 legacy=0x7fa69413ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.046 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 1 monclient: found mon.noname-c-legacy 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940a97f0 con 0x7fa6940a8a30 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2176119509 0 0) 0x7fa684001670 con 0x7fa6940a8a30 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.043+0000 7fa698a7f640 10 monclient: dump: 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.047 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 >> v1:172.21.15.154:6789/0 conn(0x7fa6940a8a30 legacy=0x7fa6941424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa680003970 0x7fa680003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): picked mon.b con 0x7fa680003970 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6800047c0 0x7fa680004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): picked mon.a con 0x7fa6800047c0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa680005180 0x7fa68000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): picked mon.c con 0x7fa680005180 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6940aaa10 con 0x7fa6800047c0 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68000c050 con 0x7fa680003970 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68000c1c0 con 0x7fa680005180 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.049 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.044+0000 7fa698a7f640 10 monclient(hunting): _finish_auth 0 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.045+0000 7fc41b57d640 10 monclient: get_monmap_and_config 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.045+0000 7fc41b57d640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.045+0000 7fc41b57d640 10 monclient: monmap: 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:46.046140+0000 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:46.046140+0000 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:46.050 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:46.051 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa680003970 0x7fa680003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fc41b57d640 1 Processor -- start 2023-12-07T07:33:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 10 monclient(hunting): get_auth_request con 0x7fa680003970 auth_method 0 2023-12-07T07:33:46.051 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:33:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fc41b57d640 1 -- start start 2023-12-07T07:33:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fc41b57d640 10 monclient: init 2023-12-07T07:33:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.046+0000 7fa699a81640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4140a8a30 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc414142bf0 0x7fc41413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient(hunting): picked mon.noname-a con 0x7fc414142bf0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4140a8e10 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4140aac10 con 0x7fc4140a8e10 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4140aaef0 con 0x7fc4140a8a30 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4140aaa10 con 0x7fc414142bf0 2023-12-07T07:33:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.047+0000 7fc41b57d640 10 monclient(hunting): authenticate will time out at 1348.425634s 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 10 monclient(hunting): handle_auth_done global_id 4345 payload 293 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 1 -- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa680005180 msgr2=0x7fa68000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa680005180 0x7fa68000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 1 -- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6800047c0 msgr2=0x7fa680004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6800047c0 0x7fa680004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 1 monclient: found mon.b 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 1 -- 172.21.15.97:0/425118689 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940a97f0 con 0x7fa680003970 2023-12-07T07:33:46.055 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa699a81640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa69bce5640 5 monclient: authenticate success, global_id 4345 2023-12-07T07:33:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.048+0000 7fa69bce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.050+0000 7fc419319640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc4140a8a30 0x7fc4141424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44736/0 (socket says 172.21.15.97:44736) 2023-12-07T07:33:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.050+0000 7fc419319640 1 -- 172.21.15.97:0/3297370251 learned_addr learned my addr 172.21.15.97:0/3297370251 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.051+0000 7fa699a81640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa680003970 0x7fa680003d70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa68400c290 tx=0x7fa6800057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcb3c118c7c90f1a server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.054+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950207523 0 0) 0x7fc4140aaef0 con 0x7fc4140a8a30 2023-12-07T07:33:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.054+0000 7fc40bfff640 10 monclient(hunting): _negotiate 2023-12-07T07:33:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.054+0000 7fc40bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.054+0000 7fc40bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.054+0000 7fc40bfff640 10 monclient(hunting): my global_id is 4348 2023-12-07T07:33:46.057 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.054+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc400003520 con 0x7fc4140a8a30 2023-12-07T07:33:46.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68402cb60 con 0x7fa680003970 2023-12-07T07:33:46.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 10 monclient: dump: 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.062 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68402ccc0 con 0x7fa680003970 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 1 -- 172.21.15.97:0/425118689 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684005b10 con 0x7fa680003970 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa698a7f640 10 monclient: dump: 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.063 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa69bce5640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:46.064 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.059+0000 7fa69bce5640 4 set_mon_vals no callback set 2023-12-07T07:33:46.064 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3923101204 0 0) 0x7fc400003520 con 0x7fc4140a8a30 2023-12-07T07:33:46.064 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 >> v2:172.21.15.97:3300/0 conn(0x7fc414142bf0 msgr2=0x7fc41413ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.064 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 --2- 172.21.15.97:0/3297370251 >> v2:172.21.15.97:3300/0 conn(0x7fc414142bf0 0x7fc41413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.065 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fa69bce5640 10 monclient: shutdown 2023-12-07T07:33:46.065 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fa69bce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.065 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fa69bce5640 1 -- 172.21.15.97:0/425118689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa680003970 msgr2=0x7fa680003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.065 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fa69bce5640 1 --2- 172.21.15.97:0/425118689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa680003970 0x7fa680003d70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa68400c290 tx=0x7fa6800057f0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.065 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fa69a282640 1 -- 172.21.15.97:0/425118689 reap_dead start 2023-12-07T07:33:46.065 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 >> v1:172.21.15.97:6789/0 conn(0x7fc4140a8e10 legacy=0x7fc41413f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.065 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.066 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fa69bce5640 1 -- 172.21.15.97:0/425118689 shutdown_connections 2023-12-07T07:33:46.066 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fa69bce5640 1 -- 172.21.15.97:0/425118689 >> 172.21.15.97:0/425118689 conn(0x7fa694061a30 msgr2=0x7fa6940b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.066 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 monclient: found mon.noname-c-legacy 2023-12-07T07:33:46.067 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.067 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2023-12-07T07:33:46.067 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4140a97f0 con 0x7fc4140a8a30 2023-12-07T07:33:46.067 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.061+0000 7fa69bce5640 1 -- 172.21.15.97:0/425118689 shutdown_connections 2023-12-07T07:33:46.069 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2176119509 0 0) 0x7fc40c0019f0 con 0x7fc4140a8a30 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: dump: 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.070 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 >> v1:172.21.15.154:6789/0 conn(0x7fc4140a8a30 legacy=0x7fc4141424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc400003970 0x7fc400003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): picked mon.c con 0x7fc400003970 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4000047c0 0x7fc400004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): picked mon.a con 0x7fc4000047c0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc40000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): picked mon.b con 0x7fc4000052a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140aaa10 con 0x7fc4000047c0 2023-12-07T07:33:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40000c190 con 0x7fc4000052a0 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40000c350 con 0x7fc400003970 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.060+0000 7fc40bfff640 10 monclient(hunting): _finish_auth 0 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.061+0000 7fc419b1a640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc40000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fc419b1a640 10 monclient(hunting): get_auth_request con 0x7fc4000052a0 auth_method 0 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fc419b1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fc419b1a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.072 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fc419b1a640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.061+0000 7fa69bce5640 1 -- 172.21.15.97:0/425118689 wait complete. 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fa69bce5640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fa69bce5640 10 monclient: monmap: 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:46.073 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fa69bce5640 1 librados: starting msgr at 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fa69bce5640 1 librados: starting objecter 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.062+0000 7fa69bce5640 1 Processor -- start 2023-12-07T07:33:46.074 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fc419b1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 1 -- start start 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 1 librados: setting wanted keys 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 1 librados: calling monclient init 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 10 monclient: init 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa680003970 0x7fa6941410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 10 monclient(hunting): picked mon.noname-c con 0x7fa680003970 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fa69bce5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa694144db0 0x7fa6941454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fc419b1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.063+0000 7fc419b1a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:33:46.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 10 monclient(hunting): picked mon.noname-b con 0x7fa694144db0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:46.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa694136b50 0x7fa694136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 10 monclient(hunting): picked mon.noname-a con 0x7fa694136b50 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941445e0 con 0x7fa694136b50 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941451f0 con 0x7fa694144db0 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa694145c20 con 0x7fa680003970 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.064+0000 7fa69bce5640 10 monclient(hunting): authenticate will time out at 1348.442696s 2023-12-07T07:33:46.077 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 10 monclient(hunting): handle_auth_done global_id 4348 payload 293 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 1 -- 172.21.15.97:0/3297370251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc400003970 msgr2=0x7fc400003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc400003970 0x7fc400003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 1 -- 172.21.15.97:0/3297370251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4000047c0 msgr2=0x7fc400004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4000047c0 0x7fc400004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 1 monclient: found mon.b 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 1 -- 172.21.15.97:0/3297370251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4140a97f0 con 0x7fc4000052a0 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.065+0000 7fc419b1a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.066+0000 7fc41b57d640 5 monclient: authenticate success, global_id 4348 2023-12-07T07:33:46.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.066+0000 7fc41b57d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.066+0000 7fc419b1a640 1 -- 172.21.15.97:0/3297370251 reap_dead start 2023-12-07T07:33:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.067+0000 7fa699280640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa694144db0 0x7fa6941454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.067+0000 7fa699280640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa694144db0 0x7fa6941454c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38684/0 (socket says 172.21.15.97:38684) 2023-12-07T07:33:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.067+0000 7fa699280640 1 -- 172.21.15.97:0/466798109 learned_addr learned my addr 172.21.15.97:0/466798109 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.067+0000 7fa699280640 10 monclient(hunting): get_auth_request con 0x7fa694144db0 auth_method 0 2023-12-07T07:33:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.067+0000 7fa699280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.067+0000 7fa699280640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.067+0000 7fa699280640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.068+0000 7fa699280640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.068+0000 7fa699280640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.068+0000 7fa699280640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.068+0000 7fc419b1a640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc40000b980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc404001620 tx=0x7fc4000057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f43bea354248023 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 1 -- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa680003970 msgr2=0x7fa6941410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa680003970 0x7fa6941410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 1 -- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa694136b50 msgr2=0x7fa694136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa694136b50 0x7fa694136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 1 monclient: found mon.noname-b 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.071+0000 7fa699280640 1 -- 172.21.15.97:0/466798109 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa694137510 con 0x7fa694144db0 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.076+0000 7fa699280640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa694144db0 0x7fa6941454c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa69000c440 tx=0x7fa69000c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.076+0000 7fa68a7fc640 10 client.?.objecter ms_handle_connect 0x7fa694144db0 2023-12-07T07:33:46.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.076+0000 7fa68a7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4040052d0 con 0x7fc4000052a0 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 10 monclient: dump: 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.084 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40402e070 con 0x7fc4000052a0 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 1 -- 172.21.15.97:0/3297370251 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404003830 con 0x7fc4000052a0 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.077+0000 7fc40bfff640 10 monclient: dump: 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.085 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.078+0000 7fc41b57d640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.078+0000 7fc41b57d640 4 set_mon_vals no callback set 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.079+0000 7fc41b57d640 10 monclient: shutdown 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.079+0000 7fc41b57d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.079+0000 7fc41b57d640 1 -- 172.21.15.97:0/3297370251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 msgr2=0x7fc40000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.079+0000 7fc41b57d640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc40000b980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc404001620 tx=0x7fc4000057f0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.079+0000 7fc41b57d640 1 -- 172.21.15.97:0/3297370251 shutdown_connections 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.079+0000 7fc41b57d640 1 --2- 172.21.15.97:0/3297370251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc40000b980 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.079+0000 7fc41b57d640 1 -- 172.21.15.97:0/3297370251 >> 172.21.15.97:0/3297370251 conn(0x7fc414061a30 msgr2=0x7fc4140b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.080+0000 7fc41b57d640 1 -- 172.21.15.97:0/3297370251 shutdown_connections 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.080+0000 7fc41b57d640 1 -- 172.21.15.97:0/3297370251 wait complete. 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.081+0000 7fc41b57d640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.081+0000 7fc41b57d640 10 monclient: monmap: 2023-12-07T07:33:46.086 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.081+0000 7fc41b57d640 1 librados: starting msgr at 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.081+0000 7fc41b57d640 1 librados: starting objecter 2023-12-07T07:33:46.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.081+0000 7fc41b57d640 1 Processor -- start 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.082+0000 7fc41b57d640 1 -- start start 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.082+0000 7fc41b57d640 1 librados: setting wanted keys 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.082+0000 7fc41b57d640 1 librados: calling monclient init 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.082+0000 7fc41b57d640 10 monclient: init 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.083+0000 7fc41b57d640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.083+0000 7fc41b57d640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.083+0000 7fc41b57d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc414143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.083+0000 7fc41b57d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4000052a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.083+0000 7fc41b57d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4140ab370 0x7fc414145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.083+0000 7fc41b57d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4140ab370 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:46.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 1 -- 172.21.15.97:0/466798109 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690005480 con 0x7fa694144db0 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: dump: 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 1 -- 172.21.15.97:0/466798109 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6900035f0 con 0x7fa694144db0 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 1 -- 172.21.15.97:0/466798109 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6900039a0 con 0x7fa694144db0 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa68a7fc640 10 monclient: dump: 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.091 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.092 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.092 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fc419319640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc414143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fc41b57d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4141d7360 0x7fc4141da0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fc41b57d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4141d7360 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fc41b57d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 5 monclient: authenticate success, global_id 4351 2023-12-07T07:33:46.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 10 monclient: _renew_subs 2023-12-07T07:33:46.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6941d6fd0 con 0x7fa694144db0 2023-12-07T07:33:46.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fc419319640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc414143050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38688/0 (socket says 172.21.15.97:38688) 2023-12-07T07:33:46.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fc419319640 1 -- 172.21.15.97:0/1318416079 learned_addr learned my addr 172.21.15.97:0/1318416079 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:46.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:46.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 10 monclient: _renew_subs 2023-12-07T07:33:46.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6941d7490 con 0x7fa694144db0 2023-12-07T07:33:46.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fa69bce5640 1 librados: init done 2023-12-07T07:33:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.084+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4141d6bf0 con 0x7fc4140ab370 2023-12-07T07:33:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc41b57d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4141d6ed0 con 0x7fc4000052a0 2023-12-07T07:33:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc41b57d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140b74e0 con 0x7fc4141d7360 2023-12-07T07:33:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc41b57d640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc41b57d640 10 monclient(hunting): authenticate will time out at 1348.463677s 2023-12-07T07:33:46.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc419319640 10 monclient(hunting): get_auth_request con 0x7fc4000052a0 auth_method 0 2023-12-07T07:33:46.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc419319640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc419319640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fc419319640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.085+0000 7fa68b7fe640 4 set_mon_vals no callback set 2023-12-07T07:33:46.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.086+0000 7fa69bce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.086+0000 7fa69bce5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.086+0000 7fa69bce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.086+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67c005040 con 0x7fa694144db0 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.092+0000 7fadb24ff640 10 monclient: get_monmap_and_config 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.092+0000 7fadb24ff640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.092+0000 7fadb24ff640 10 monclient: monmap: 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:46.093683+0000 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:46.093683+0000 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:46.099 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.093+0000 7fadb24ff640 1 Processor -- start 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.093+0000 7fadb24ff640 1 -- start start 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.093+0000 7fadb24ff640 10 monclient: init 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.094+0000 7fadb24ff640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.094+0000 7fadb24ff640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada40c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): picked mon.noname-b con 0x7fada4038d70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fada4039150 0x7fada40c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): picked mon.noname-a con 0x7fada4039150 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fada40c5760 0x7fada40c8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): picked mon.noname-c con 0x7fada40c5760 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada4037570 con 0x7fada4039150 2023-12-07T07:33:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada40c7e60 con 0x7fada40c5760 2023-12-07T07:33:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada4035860 con 0x7fada4038d70 2023-12-07T07:33:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.095+0000 7fadb24ff640 10 monclient(hunting): authenticate will time out at 1348.473824s 2023-12-07T07:33:46.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.103+0000 7fa68a7fc640 1 -- 172.21.15.97:0/466798109 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa690003b00 con 0x7fa694144db0 2023-12-07T07:33:46.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.104+0000 7fa68a7fc640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa67806c830 0x7fa67806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.104+0000 7fa699a81640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa67806c830 0x7fa67806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.104+0000 7fa699a81640 10 monclient: get_auth_request con 0x7fa67806c830 auth_method 0 2023-12-07T07:33:46.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.105+0000 7fa699a81640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa67806c830 0x7fa67806ecf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa684034000 tx=0x7fa684007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.105+0000 7fa68a7fc640 10 client.4351.objecter ms_handle_connect 0x7fa67806c830 2023-12-07T07:33:46.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.105+0000 7fa68a7fc640 1 -- 172.21.15.97:0/466798109 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa690057ee0 con 0x7fa694144db0 2023-12-07T07:33:46.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.105+0000 7fa68a7fc640 10 client.4351.objecter ms_dispatch 0x7fa694000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:46.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.105+0000 7fa68a7fc640 3 client.4351.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:46.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.105+0000 7fa68a7fc640 3 client.4351.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:46.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.105+0000 7fa68a7fc640 20 client.4351.objecter dump_active .. 0 homeless 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.112+0000 7f93e556c640 10 monclient: get_monmap_and_config 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.112+0000 7f93e556c640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.112+0000 7f93e556c640 10 monclient: monmap: 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:46.113499+0000 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:46.113499+0000 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.115 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.113+0000 7f93e556c640 1 Processor -- start 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.113+0000 7f93e556c640 1 -- start start 2023-12-07T07:33:46.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.113+0000 7f93e556c640 10 monclient: init 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.113+0000 7f93e556c640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.113+0000 7f93e556c640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.113+0000 7f93e556c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93e00a8a30 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.113+0000 7f93e556c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e0142bf0 0x7f93e013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 10 monclient(hunting): picked mon.noname-b con 0x7f93e0142bf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f93e00a8e10 0x7f93e013f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 10 monclient(hunting): picked mon.noname-a con 0x7f93e00a8e10 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e00aac10 con 0x7f93e00a8a30 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e00aaa10 con 0x7f93e00a8e10 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00aaef0 con 0x7f93e0142bf0 2023-12-07T07:33:46.119 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.119 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.114+0000 7f93e556c640 10 monclient(hunting): authenticate will time out at 1348.492367s 2023-12-07T07:33:46.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.213+0000 7feb7f221640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T07:33:46.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.213+0000 7feb7f221640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T07:33:46.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.213+0000 7feb7f221640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.214+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7feb780afa30 con 0x7feb781454a0 2023-12-07T07:33:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.251+0000 7f3181ffb640 1 -- 172.21.15.97:0/3933470383 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3184007070 con 0x7f318c142bf0 2023-12-07T07:33:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.251+0000 7f3181ffb640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f317006c820 0x7f317006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.252+0000 7f319123e640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f317006c820 0x7f317006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.252+0000 7f319123e640 10 monclient: get_auth_request con 0x7f317006c820 auth_method 0 2023-12-07T07:33:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.252+0000 7f319123e640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f317006c820 0x7f317006ece0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3188001560 tx=0x7f3188007990 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.252+0000 7f3181ffb640 10 client.4311.objecter ms_handle_connect 0x7f317006c820 2023-12-07T07:33:46.258 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.256+0000 7f3181ffb640 1 -- 172.21.15.97:0/3933470383 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3184058f70 con 0x7f318c142bf0 2023-12-07T07:33:46.258 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.256+0000 7f3181ffb640 10 client.4311.objecter ms_dispatch 0x7f318c000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:46.258 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.256+0000 7f3181ffb640 3 client.4311.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:46.258 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.256+0000 7f3181ffb640 3 client.4311.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:46.258 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.256+0000 7f3181ffb640 20 client.4311.objecter dump_active .. 0 homeless 2023-12-07T07:33:46.288 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.286+0000 7f70d77fe640 1 -- 172.21.15.97:0/3811515102 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70e8055240 con 0x7f70f00ae350 2023-12-07T07:33:46.289 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.286+0000 7f70d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.289 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.286+0000 7f70d77fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:46.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.288+0000 7fadb14fd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada40c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.288+0000 7fc419b1a640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4141d7360 0x7fc4141da0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada40c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38224/0 (socket says 172.21.15.97:38224) 2023-12-07T07:33:46.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 1 -- 172.21.15.97:0/2734189389 learned_addr learned my addr 172.21.15.97:0/2734189389 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:46.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 10 monclient(hunting): get_auth_request con 0x7fada4038d70 auth_method 0 2023-12-07T07:33:46.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fc419b1a640 10 monclient(hunting): get_auth_request con 0x7fc4141d7360 auth_method 0 2023-12-07T07:33:46.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fc419b1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fc419b1a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fc419b1a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.289+0000 7fadb14fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:46.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.290+0000 7fc419b1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.290+0000 7fc419b1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.290+0000 7fc419b1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:46.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2023-12-07T07:33:46.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 1 -- 172.21.15.97:0/2734189389 >> v2:172.21.15.154:3300/0 conn(0x7fada40c5760 msgr2=0x7fada40c8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 1 --2- 172.21.15.97:0/2734189389 >> v2:172.21.15.154:3300/0 conn(0x7fada40c5760 0x7fada40c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 1 -- 172.21.15.97:0/2734189389 >> v2:172.21.15.97:3300/0 conn(0x7fada4039150 msgr2=0x7fada40c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 1 --2- 172.21.15.97:0/2734189389 >> v2:172.21.15.97:3300/0 conn(0x7fada4039150 0x7fada40c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 1 monclient: found mon.noname-b 2023-12-07T07:33:46.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:46.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.292+0000 7fadb14fd640 1 -- 172.21.15.97:0/2734189389 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada4036060 con 0x7fada4038d70 2023-12-07T07:33:46.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2023-12-07T07:33:46.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 1 -- 172.21.15.97:0/1318416079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 msgr2=0x7fc414143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc414143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 1 -- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4140ab370 msgr2=0x7fc414145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4140ab370 0x7fc414145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 1 monclient: found mon.noname-c 2023-12-07T07:33:46.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:46.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.296+0000 7fc419b1a640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414146080 con 0x7fc4141d7360 2023-12-07T07:33:46.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.299+0000 7fadb14fd640 1 --2- 172.21.15.97:0/2734189389 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada40c5220 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fada8009870 tx=0x7fada802b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7724633a8f4d60dd server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.301+0000 7fc419b1a640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4141d7360 0x7fc4141da0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc414142fd0 tx=0x7fc404000fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.301+0000 7fc409ffb640 10 client.?.objecter ms_handle_connect 0x7fc4141d7360 2023-12-07T07:33:46.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.301+0000 7fc409ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 1 -- 172.21.15.97:0/1318416079 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40403a070 con 0x7fc4141d7360 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: dump: 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.308 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 1 -- 172.21.15.97:0/1318416079 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404007250 con 0x7fc4141d7360 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 5 monclient: authenticate success, global_id 4310 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 1 -- 172.21.15.97:0/1318416079 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404007620 con 0x7fc4141d7360 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc409ffb640 10 monclient: dump: 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.309 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 10 monclient: _renew_subs 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc414146270 con 0x7fc4141d7360 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc40b7fe640 4 set_mon_vals no callback set 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 10 monclient: _renew_subs 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:46.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4141d79c0 con 0x7fc4141d7360 2023-12-07T07:33:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.305+0000 7fc41b57d640 1 librados: init done 2023-12-07T07:33:46.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.306+0000 7f93de7fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e0142bf0 0x7f93e013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.306+0000 7fc41b57d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.306+0000 7fc41b57d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.306+0000 7fc41b57d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.306+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc414067ac0 con 0x7fc4141d7360 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.310+0000 7f93de7fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e0142bf0 0x7f93e013ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38232/0 (socket says 172.21.15.97:38232) 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.310+0000 7f93de7fc640 1 -- 172.21.15.97:0/2443723695 learned_addr learned my addr 172.21.15.97:0/2443723695 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.310+0000 7f93de7fc640 10 monclient(hunting): get_auth_request con 0x7f93e0142bf0 auth_method 0 2023-12-07T07:33:46.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.310+0000 7f93de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.310+0000 7f93de7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.310+0000 7f93de7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.346+0000 7fa68a7fc640 1 -- 172.21.15.97:0/466798109 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa690053b20 con 0x7fa694144db0 2023-12-07T07:33:46.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.346+0000 7fa68a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.346+0000 7fa68a7fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:46.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.357+0000 7feb757fa640 1 -- 172.21.15.97:0/2422480110 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb70017070 con 0x7feb781454a0 2023-12-07T07:33:46.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.357+0000 7feb757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T07:33:46.359 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.357+0000 7feb757fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:46.360 INFO:teuthology.orchestra.run.smithi097.stdout:73014444035 2023-12-07T07:33:46.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 10 librados: watch_flush enter 2023-12-07T07:33:46.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 10 librados: watch_flush exit 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 20 client.4342.objecter shutdown clearing up homeless session... 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 10 client.4342.objecter successfully canceled tick 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feb5c06c7d0 msgr2=0x7feb5c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feb5c06c7d0 0x7feb5c06ec90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feb6c036000 tx=0x7feb6c03c040 comp rx=0 tx=0).stop 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 10 monclient: shutdown 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb781454a0 msgr2=0x7feb7813b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb781454a0 0x7feb7813b670 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb7000ca20 tx=0x7feb7000cee0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 shutdown_connections 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feb5c06c7d0 0x7feb5c06ec90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb781454a0 0x7feb7813b670 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb780a8e10 0x7feb780b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 --2- 172.21.15.97:0/2422480110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb780a8a30 0x7feb780af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 >> 172.21.15.97:0/2422480110 conn(0x7feb78061a30 msgr2=0x7feb7813cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 shutdown_connections 2023-12-07T07:33:46.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 -- 172.21.15.97:0/2422480110 wait complete. 2023-12-07T07:33:46.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.360+0000 7feb7f221640 1 librados: shutdown 2023-12-07T07:33:46.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444035 for osd.2 2023-12-07T07:33:46.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.476+0000 7f70d57fa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:46.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.476+0000 7f70d57fa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:46.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.476+0000 7f70d57fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:46.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.476+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f70f01408a0 con 0x7f70f00ae350 2023-12-07T07:33:46.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.516+0000 7f3181ffb640 1 -- 172.21.15.97:0/3933470383 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3184054bb0 con 0x7f318c142bf0 2023-12-07T07:33:46.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.516+0000 7f3181ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.516+0000 7f3181ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:46.537 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.535+0000 7fa69bce5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.535+0000 7fa69bce5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.535+0000 7fa69bce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.535+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa67c0055c0 con 0x7fa694144db0 2023-12-07T07:33:46.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.539+0000 7fa68a7fc640 1 -- 172.21.15.97:0/466798109 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa690017090 con 0x7fa694144db0 2023-12-07T07:33:46.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.539+0000 7fa68a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:46.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.539+0000 7fa68a7fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:46.542 INFO:teuthology.orchestra.run.smithi097.stdout:55834574853 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 10 librados: watch_flush enter 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 10 librados: watch_flush exit 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 20 client.4351.objecter shutdown clearing up homeless session... 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 10 client.4351.objecter successfully canceled tick 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa67806c830 msgr2=0x7fa67806ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa67806c830 0x7fa67806ecf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa684034000 tx=0x7fa684007000 comp rx=0 tx=0).stop 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 10 monclient: shutdown 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa694144db0 msgr2=0x7fa6941454c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa694144db0 0x7fa6941454c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa69000c440 tx=0x7fa69000c900 comp rx=0 tx=0).stop 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 shutdown_connections 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa67806c830 0x7fa67806ecf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa694136b50 0x7fa694136fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa694144db0 0x7fa6941454c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 --2- 172.21.15.97:0/466798109 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa680003970 0x7fa6941410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 >> 172.21.15.97:0/466798109 conn(0x7fa694061a30 msgr2=0x7fa6940b6150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 shutdown_connections 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.541+0000 7fa69bce5640 1 -- 172.21.15.97:0/466798109 wait complete. 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.542+0000 7fa69bce5640 1 librados: shutdown 2023-12-07T07:33:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.543+0000 7f92acff9640 1 -- 172.21.15.97:0/3334993634 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92a8054ac0 con 0x7f92b01454a0 2023-12-07T07:33:46.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.543+0000 7f92acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.543+0000 7f92acff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:46.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2023-12-07T07:33:46.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.558+0000 7f93df7fe640 1 --2- 172.21.15.97:0/2443723695 >> v2:172.21.15.97:3300/0 conn(0x7f93e00a8e10 0x7f93e013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.558+0000 7f93df7fe640 10 monclient(hunting): get_auth_request con 0x7f93e00a8e10 auth_method 0 2023-12-07T07:33:46.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.558+0000 7f93df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.558+0000 7f93df7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.558+0000 7f93df7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.564+0000 7f93df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.564+0000 7f93df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.564+0000 7f93df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7fc409ffb640 1 -- 172.21.15.97:0/1318416079 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc404030030 con 0x7fc4141d7360 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 1 -- 172.21.15.97:0/2443723695 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e0142bf0 msgr2=0x7f93e013ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 1 --2- 172.21.15.97:0/2443723695 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e0142bf0 0x7f93e013ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 1 -- 172.21.15.97:0/2443723695 >> v1:172.21.15.97:6789/0 conn(0x7f93e00a8a30 legacy=0x7f93e01424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 1 monclient: found mon.noname-a 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.566+0000 7f93df7fe640 1 -- 172.21.15.97:0/2443723695 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e00a97f0 con 0x7f93e00a8e10 2023-12-07T07:33:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.567+0000 7fc409ffb640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc3f806c820 0x7fc3f806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.567+0000 7fc419319640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc3f806c820 0x7fc3f806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.567+0000 7fc419319640 10 monclient: get_auth_request con 0x7fc3f806c820 auth_method 0 2023-12-07T07:33:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.567+0000 7fc419319640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc3f806c820 0x7fc3f806ece0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0021a0 tx=0x7fc40c005b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.567+0000 7fc409ffb640 10 client.4310.objecter ms_handle_connect 0x7fc3f806c820 2023-12-07T07:33:46.577 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.573+0000 7f93df7fe640 1 --2- 172.21.15.97:0/2443723695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e013f230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009d90 tx=0x7f93d802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e839334fe5ce8577 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 1 -- 172.21.15.97:0/2443723695 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d802cc60 con 0x7f93e00a8e10 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: dump: 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 1 -- 172.21.15.97:0/2443723695 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d802cdc0 con 0x7f93e00a8e10 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93e556c640 5 monclient: authenticate success, global_id 4314 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93e556c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93e556c640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93e556c640 4 set_mon_vals no callback set 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 1 -- 172.21.15.97:0/2443723695 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8034660 con 0x7f93e00a8e10 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.580+0000 7f93ddffb640 10 monclient: dump: 2023-12-07T07:33:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 10 monclient: shutdown 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 -- 172.21.15.97:0/2443723695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 msgr2=0x7f93e013f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 --2- 172.21.15.97:0/2443723695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e013f230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009d90 tx=0x7f93d802beb0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 -- 172.21.15.97:0/2443723695 shutdown_connections 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 --2- 172.21.15.97:0/2443723695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e013f230 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 --2- 172.21.15.97:0/2443723695 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e0142bf0 0x7f93e013ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 -- 172.21.15.97:0/2443723695 >> 172.21.15.97:0/2443723695 conn(0x7f93e0061a30 msgr2=0x7f93e00b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 -- 172.21.15.97:0/2443723695 shutdown_connections 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.581+0000 7f93e556c640 1 -- 172.21.15.97:0/2443723695 wait complete. 2023-12-07T07:33:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.582+0000 7f93e556c640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.582+0000 7f93e556c640 10 monclient: monmap: 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.582+0000 7f93e556c640 1 librados: starting msgr at 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.582+0000 7f93e556c640 1 librados: starting objecter 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.583+0000 7f93e556c640 1 Processor -- start 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.583+0000 7f93e556c640 1 -- start start 2023-12-07T07:33:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.583+0000 7f93e556c640 1 librados: setting wanted keys 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.583+0000 7f93e556c640 1 librados: calling monclient init 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.583+0000 7f93e556c640 10 monclient: init 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e00a8a30 0x7f93e01d7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): picked mon.noname-c con 0x7f93e00a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e00ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): picked mon.noname-a con 0x7f93e00a8e10 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93e0142bf0 0x7f93e00acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): picked mon.noname-b con 0x7f93e0142bf0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00abea0 con 0x7f93e00a8e10 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00ae5e0 con 0x7f93e0142bf0 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e01d79a0 con 0x7f93e00a8a30 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93e556c640 10 monclient(hunting): authenticate will time out at 1348.962624s 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93de7fc640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e00ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.584+0000 7f93df7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93e0142bf0 0x7f93e00acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e00ac330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39026/0 (socket says 172.21.15.97:39026) 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93e0142bf0 0x7f93e00acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38704/0 (socket says 172.21.15.97:38704) 2023-12-07T07:33:46.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 1 -- 172.21.15.97:0/264932474 learned_addr learned my addr 172.21.15.97:0/264932474 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 10 monclient(hunting): get_auth_request con 0x7f93e00a8e10 auth_method 0 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 10 monclient(hunting): get_auth_request con 0x7f93e0142bf0 auth_method 0 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.585+0000 7f93de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 1 -- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e00a8a30 msgr2=0x7f93e01d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e00a8a30 0x7f93e01d7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 1 -- 172.21.15.97:0/264932474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93e0142bf0 msgr2=0x7f93e00acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93e0142bf0 0x7f93e00acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 1 monclient: found mon.noname-a 2023-12-07T07:33:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7f93de7fc640 1 -- 172.21.15.97:0/264932474 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8009a70 con 0x7f93e00a8e10 2023-12-07T07:33:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 1 -- 172.21.15.97:0/1318416079 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc404078cb0 con 0x7fc4141d7360 2023-12-07T07:33:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 10 client.4310.objecter ms_dispatch 0x7fc414000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:46.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 3 client.4310.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 3 client.4310.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 20 client.4310.objecter dump_active .. 0 homeless 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 1 -- 172.21.15.97:0/1318416079 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4040748f0 con 0x7fc4141d7360 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.588+0000 7fc409ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.591+0000 7f93de7fc640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e00ac330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f93d400d830 tx=0x7f93d400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.591+0000 7f93c77fe640 10 client.?.objecter ms_handle_connect 0x7f93e00a8e10 2023-12-07T07:33:46.595 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.591+0000 7f93c77fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:46.598 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 1 -- 172.21.15.97:0/264932474 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40096a0 con 0x7f93e00a8e10 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: dump: 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 1 -- 172.21.15.97:0/264932474 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d4009800 con 0x7f93e00a8e10 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 1 -- 172.21.15.97:0/264932474 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4009bb0 con 0x7f93e00a8e10 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93c77fe640 10 monclient: dump: 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.600 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 5 monclient: authenticate success, global_id 4320 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93dd7fa640 4 set_mon_vals no callback set 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 10 monclient: _renew_subs 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e00ac990 con 0x7f93e00a8e10 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 10 monclient: _renew_subs 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e00b15a0 con 0x7f93e00a8e10 2023-12-07T07:33:46.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7f93e556c640 1 librados: init done 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 1 -- 172.21.15.97:0/2734189389 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada80317e0 con 0x7fada4038d70 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: dump: 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.602 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 1 -- 172.21.15.97:0/2734189389 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada8031940 con 0x7fada4038d70 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 1 -- 172.21.15.97:0/2734189389 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8031cd0 con 0x7fada4038d70 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:46.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fada3fff640 10 monclient: dump: 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fadb24ff640 5 monclient: authenticate success, global_id 4307 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fadb24ff640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.596+0000 7fadb24ff640 4 set_mon_vals no callback set 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 10 monclient: shutdown 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 1 -- 172.21.15.97:0/2734189389 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 msgr2=0x7fada40c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 1 --2- 172.21.15.97:0/2734189389 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada40c5220 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fada8009870 tx=0x7fada802b880 comp rx=0 tx=0).stop 2023-12-07T07:33:46.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7f93e556c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7f93e556c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.605 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7f93e556c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:46.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 1 -- 172.21.15.97:0/2734189389 shutdown_connections 2023-12-07T07:33:46.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 1 --2- 172.21.15.97:0/2734189389 >> v2:172.21.15.97:3300/0 conn(0x7fada4039150 0x7fada40c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 1 --2- 172.21.15.97:0/2734189389 >> v2:172.21.15.154:3300/0 conn(0x7fada40c5760 0x7fada40c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7fadb24ff640 1 --2- 172.21.15.97:0/2734189389 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada40c5220 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.598+0000 7fadb24ff640 1 -- 172.21.15.97:0/2734189389 >> 172.21.15.97:0/2734189389 conn(0x7fada40065d0 msgr2=0x7fada40069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.606 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.597+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e0067ac0 con 0x7f93e00a8e10 2023-12-07T07:33:46.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.598+0000 7fadb24ff640 1 -- 172.21.15.97:0/2734189389 shutdown_connections 2023-12-07T07:33:46.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.598+0000 7fadb24ff640 1 -- 172.21.15.97:0/2734189389 wait complete. 2023-12-07T07:33:46.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 10 monclient: build_initial_monmap 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 10 monclient: monmap: 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 1 librados: starting msgr at 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 1 librados: starting objecter 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 1 Processor -- start 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 1 -- start start 2023-12-07T07:33:46.609 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 1 librados: setting wanted keys 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 1 librados: calling monclient init 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.599+0000 7fadb24ff640 10 monclient: init 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada4032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): picked mon.noname-c con 0x7fada4038d70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c5760 0x7fada4032980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): picked mon.noname-b con 0x7fada40c5760 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fada402e890 0x7fada402ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): picked mon.noname-a con 0x7fada402e890 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada402df20 con 0x7fada402e890 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada402e190 con 0x7fada40c5760 2023-12-07T07:33:46.610 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada4031440 con 0x7fada4038d70 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.600+0000 7fadb24ff640 10 monclient(hunting): authenticate will time out at 1348.979262s 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.601+0000 7fadb0cfc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c5760 0x7fada4032980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.601+0000 7fadb0cfc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c5760 0x7fada4032980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38720/0 (socket says 172.21.15.97:38720) 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.601+0000 7fadb0cfc640 1 -- 172.21.15.97:0/4189246836 learned_addr learned my addr 172.21.15.97:0/4189246836 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.601+0000 7fadb0cfc640 10 monclient(hunting): get_auth_request con 0x7fada40c5760 auth_method 0 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.601+0000 7fadb0cfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.601+0000 7fadb0cfc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.601+0000 7fadb0cfc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.602+0000 7fadb0cfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.602+0000 7fadb0cfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.602+0000 7fadb0cfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 1 -- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 msgr2=0x7fada4032280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.611 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada4032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 1 -- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fada402e890 msgr2=0x7fada402ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fada402e890 0x7fada402ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 1 monclient: found mon.noname-b 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.604+0000 7fadb0cfc640 1 -- 172.21.15.97:0/4189246836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada80094d0 con 0x7fada40c5760 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.606+0000 7f70d77fe640 1 -- 172.21.15.97:0/3811515102 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f70e8019070 con 0x7f70f00ae350 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.606+0000 7f70d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:46.612 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.606+0000 7f70d77fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:46.613 INFO:teuthology.orchestra.run.smithi097.stdout:73014444036 2023-12-07T07:33:46.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.607+0000 7fadb0cfc640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c5760 0x7fada4032980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00b4a0 tx=0x7fad9c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.607+0000 7fada1ffb640 10 client.?.objecter ms_handle_connect 0x7fada40c5760 2023-12-07T07:33:46.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.607+0000 7fada1ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.608+0000 7f70d57fa640 10 librados: watch_flush enter 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.608+0000 7f70d57fa640 10 librados: watch_flush exit 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.608+0000 7f70d57fa640 20 client.4301.objecter shutdown clearing up homeless session... 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.608+0000 7f70d57fa640 10 client.4301.objecter successfully canceled tick 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.608+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f70dc06c7d0 msgr2=0x7f70dc06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f70dc06c7d0 0x7f70dc06ec90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f70e00015b0 tx=0x7f70e0007990 comp rx=0 tx=0).stop 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 10 monclient: shutdown 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 msgr2=0x7f70f01d7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f01d7820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b970 tx=0x7f70e800be30 comp rx=0 tx=0).stop 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 shutdown_connections 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f70dc06c7d0 0x7f70dc06ec90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70f00ae350 0x7f70f01d7820 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70f0072330 0x7f70f011fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 --2- 172.21.15.97:0/3811515102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f006f0f0 0x7f70f011ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 >> 172.21.15.97:0/3811515102 conn(0x7f70f00b3e30 msgr2=0x7f70f013b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 shutdown_connections 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 -- 172.21.15.97:0/3811515102 wait complete. 2023-12-07T07:33:46.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.609+0000 7f70d57fa640 1 librados: shutdown 2023-12-07T07:33:46.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 1 -- 172.21.15.97:0/4189246836 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c010610 con 0x7fada40c5760 2023-12-07T07:33:46.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.616 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: dump: 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 1 -- 172.21.15.97:0/4189246836 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c010770 con 0x7fada40c5760 2023-12-07T07:33:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 1 -- 172.21.15.97:0/4189246836 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c010b20 con 0x7fada40c5760 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada1ffb640 10 monclient: dump: 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:46.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.610+0000 7fada37fe640 4 set_mon_vals no callback set 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.611+0000 7fadb24ff640 5 monclient: authenticate success, global_id 4360 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.611+0000 7fadb24ff640 10 monclient: _renew_subs 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.611+0000 7fadb24ff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.611+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada4032fa0 con 0x7fada40c5760 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.612+0000 7fadb24ff640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.612+0000 7fadb24ff640 10 monclient: _renew_subs 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.612+0000 7fadb24ff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.612+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada402f280 con 0x7fada40c5760 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.612+0000 7fadb24ff640 1 librados: init done 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.613+0000 7fadb24ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.613+0000 7fadb24ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.613+0000 7fadb24ff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:46.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.613+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada40331c0 con 0x7fada40c5760 2023-12-07T07:33:46.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444036 for osd.1 2023-12-07T07:33:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.621+0000 7f93c77fe640 1 -- 172.21.15.97:0/264932474 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f93d4014410 con 0x7f93e00a8e10 2023-12-07T07:33:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.622+0000 7f93c77fe640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f93cc06c850 0x7f93cc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.622+0000 7f93deffd640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f93cc06c850 0x7f93cc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.622+0000 7f93deffd640 10 monclient: get_auth_request con 0x7f93cc06c850 auth_method 0 2023-12-07T07:33:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.623+0000 7f93c77fe640 10 client.4320.objecter ms_handle_connect 0x7f93cc06c850 2023-12-07T07:33:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.623+0000 7f93deffd640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f93cc06c850 0x7f93cc06ed10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f93d0000f60 tx=0x7f93d00079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.625+0000 7f93c77fe640 1 -- 172.21.15.97:0/264932474 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f93d40599b0 con 0x7f93e00a8e10 2023-12-07T07:33:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.625+0000 7f93c77fe640 10 client.4320.objecter ms_dispatch 0x7f93e0000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.625+0000 7f93c77fe640 3 client.4320.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.625+0000 7f93c77fe640 3 client.4320.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.625+0000 7f93c77fe640 20 client.4320.objecter dump_active .. 0 homeless 2023-12-07T07:33:46.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.724+0000 7f31934a2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T07:33:46.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.724+0000 7f31934a2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T07:33:46.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.724+0000 7f31934a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:46.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.724+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f318c075b60 con 0x7f318c142bf0 2023-12-07T07:33:46.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.760+0000 7f92967fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T07:33:46.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.760+0000 7f92967fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T07:33:46.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.760+0000 7f92967fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:46.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.760+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f92b00b0680 con 0x7f92b01454a0 2023-12-07T07:33:46.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.793+0000 7fc41b57d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:46.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.793+0000 7fc41b57d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:46.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.793+0000 7fc41b57d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:46.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.793+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc414075b60 con 0x7fc4141d7360 2023-12-07T07:33:46.798 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.796+0000 7fc409ffb640 1 -- 172.21.15.97:0/1318416079 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc404077f10 con 0x7fc4141d7360 2023-12-07T07:33:46.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.796+0000 7fc409ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:46.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.796+0000 7fc409ffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:46.799 INFO:teuthology.orchestra.run.smithi097.stdout:55834574851 2023-12-07T07:33:46.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 10 librados: watch_flush enter 2023-12-07T07:33:46.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 10 librados: watch_flush exit 2023-12-07T07:33:46.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 20 client.4310.objecter shutdown clearing up homeless session... 2023-12-07T07:33:46.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 10 client.4310.objecter successfully canceled tick 2023-12-07T07:33:46.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc3f806c820 msgr2=0x7fc3f806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc3f806c820 0x7fc3f806ece0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0021a0 tx=0x7fc40c005b70 comp rx=0 tx=0).stop 2023-12-07T07:33:46.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 10 monclient: shutdown 2023-12-07T07:33:46.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4141d7360 msgr2=0x7fc4141da0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.798+0000 7fc41b57d640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4141d7360 0x7fc4141da0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc414142fd0 tx=0x7fc404000fb0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 shutdown_connections 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc3f806c820 0x7fc3f806ece0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4141d7360 0x7fc4141da0f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4140ab370 0x7fc414145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 --2- 172.21.15.97:0/1318416079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4000052a0 0x7fc414143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 >> 172.21.15.97:0/1318416079 conn(0x7fc414061a30 msgr2=0x7fc41413f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 shutdown_connections 2023-12-07T07:33:46.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 -- 172.21.15.97:0/1318416079 wait complete. 2023-12-07T07:33:46.804 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.799+0000 7fc41b57d640 1 librados: shutdown 2023-12-07T07:33:46.812 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.4 2023-12-07T07:33:46.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.853+0000 7fada1ffb640 1 -- 172.21.15.97:0/4189246836 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fad9c010c80 con 0x7fada40c5760 2023-12-07T07:33:46.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.853+0000 7fada1ffb640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fad9006c820 0x7fad9006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:46.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.853+0000 7fadb14fd640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fad9006c820 0x7fad9006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:46.856 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.853+0000 7fadb14fd640 10 monclient: get_auth_request con 0x7fad9006c820 auth_method 0 2023-12-07T07:33:46.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.854+0000 7fada1ffb640 10 client.4360.objecter ms_handle_connect 0x7fad9006c820 2023-12-07T07:33:46.857 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.854+0000 7fadb14fd640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fad9006c820 0x7fad9006ece0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fada8036000 tx=0x7fada8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:46.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.856+0000 7fada1ffb640 1 -- 172.21.15.97:0/4189246836 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fad9c058ab0 con 0x7fada40c5760 2023-12-07T07:33:46.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.856+0000 7fada1ffb640 10 client.4360.objecter ms_dispatch 0x7fadac000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:46.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.856+0000 7fada1ffb640 3 client.4360.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:46.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.856+0000 7fada1ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:46.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.856+0000 7fada1ffb640 20 client.4360.objecter dump_active .. 0 homeless 2023-12-07T07:33:46.867 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.865+0000 7fada1ffb640 1 -- 172.21.15.97:0/4189246836 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad9c0546f0 con 0x7fada40c5760 2023-12-07T07:33:46.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.865+0000 7fada1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.865+0000 7fada1ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:46.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.877+0000 7f93c77fe640 1 -- 172.21.15.97:0/264932474 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93d40555f0 con 0x7f93e00a8e10 2023-12-07T07:33:46.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.877+0000 7f93c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:46.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.877+0000 7f93c77fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:46.952 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.950+0000 7f3181ffb640 1 -- 172.21.15.97:0/3933470383 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f31840581d0 con 0x7f318c142bf0 2023-12-07T07:33:46.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.950+0000 7f3181ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T07:33:46.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.950+0000 7f3181ffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:46.953 INFO:teuthology.orchestra.run.smithi097.stdout:68719476740 2023-12-07T07:33:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 10 librados: watch_flush enter 2023-12-07T07:33:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 10 librados: watch_flush exit 2023-12-07T07:33:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 20 client.4311.objecter shutdown clearing up homeless session... 2023-12-07T07:33:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 10 client.4311.objecter successfully canceled tick 2023-12-07T07:33:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f317006c820 msgr2=0x7f317006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f317006c820 0x7f317006ece0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3188001560 tx=0x7f3188007990 comp rx=0 tx=0).stop 2023-12-07T07:33:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 10 monclient: shutdown 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 msgr2=0x7f318c1453e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c1453e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f318400cb70 tx=0x7f318400a430 comp rx=0 tx=0).stop 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.953+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 shutdown_connections 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f317006c820 0x7f317006ece0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f318c142bf0 0x7f318c1453e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318c0a8e10 0x7f318c1389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 --2- 172.21.15.97:0/3933470383 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f318c0a8a30 0x7f318c144e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 >> 172.21.15.97:0/3933470383 conn(0x7f318c061a30 msgr2=0x7f318c13e290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 shutdown_connections 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 -- 172.21.15.97:0/3933470383 wait complete. 2023-12-07T07:33:46.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.954+0000 7f31934a2640 1 librados: shutdown 2023-12-07T07:33:46.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.964+0000 7f92acff9640 1 -- 172.21.15.97:0/3334993634 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f92a8019020 con 0x7f92b01454a0 2023-12-07T07:33:46.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.964+0000 7f92acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T07:33:46.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.964+0000 7f92acff9640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:46.967 INFO:teuthology.orchestra.run.smithi097.stdout:55834574852 2023-12-07T07:33:46.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.967+0000 7f92967fc640 10 librados: watch_flush enter 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.967+0000 7f92967fc640 10 librados: watch_flush exit 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.967+0000 7f92967fc640 20 client.4308.objecter shutdown clearing up homeless session... 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.967+0000 7f92967fc640 10 client.4308.objecter successfully canceled tick 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f929c06c820 msgr2=0x7f929c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f929c06c820 0x7f929c06ece0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009710 tx=0x7f92a0006c90 comp rx=0 tx=0).stop 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 10 monclient: shutdown 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b01454a0 msgr2=0x7f92b00b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b01454a0 0x7f92b00b0ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b970 tx=0x7f92a800be30 comp rx=0 tx=0).stop 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 shutdown_connections 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f929c06c820 0x7f929c06ece0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b01454a0 0x7f92b00b0ef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92b00a8e10 0x7f92b00b09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 --2- 172.21.15.97:0/3334993634 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92b00a8a30 0x7f92b00ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:46.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 >> 172.21.15.97:0/3334993634 conn(0x7f92b0061a30 msgr2=0x7f92b013cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:46.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 shutdown_connections 2023-12-07T07:33:46.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 -- 172.21.15.97:0/3334993634 wait complete. 2023-12-07T07:33:46.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:46.968+0000 7f92967fc640 1 librados: shutdown 2023-12-07T07:33:46.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.0 2023-12-07T07:33:46.982 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2023-12-07T07:33:47.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.057+0000 7fadb24ff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:47.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.057+0000 7fadb24ff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:47.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.057+0000 7fadb24ff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:47.060 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.057+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fada40085f0 con 0x7fada40c5760 2023-12-07T07:33:47.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.069+0000 7f93e556c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T07:33:47.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.069+0000 7f93e556c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T07:33:47.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.069+0000 7f93e556c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.071 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.069+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f93e0075b60 con 0x7f93e00a8e10 2023-12-07T07:33:47.075 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.073+0000 7f93c77fe640 1 -- 172.21.15.97:0/264932474 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f93d4058c10 con 0x7f93e00a8e10 2023-12-07T07:33:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.073+0000 7f93c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T07:33:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.073+0000 7f93c77fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:47.076 INFO:teuthology.orchestra.run.smithi097.stdout:60129542148 2023-12-07T07:33:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.075+0000 7f93e556c640 10 librados: watch_flush enter 2023-12-07T07:33:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.075+0000 7f93e556c640 10 librados: watch_flush exit 2023-12-07T07:33:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.075+0000 7f93e556c640 20 client.4320.objecter shutdown clearing up homeless session... 2023-12-07T07:33:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.075+0000 7f93e556c640 10 client.4320.objecter successfully canceled tick 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.075+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f93cc06c850 msgr2=0x7f93cc06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.075+0000 7f93e556c640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f93cc06c850 0x7f93cc06ed10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f93d0000f60 tx=0x7f93d00079b0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 10 monclient: shutdown 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 msgr2=0x7f93e00ac330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e00ac330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f93d400d830 tx=0x7f93d400dcf0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 shutdown_connections 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f93cc06c850 0x7f93cc06ed10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93e0142bf0 0x7f93e00acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f93e00a8e10 0x7f93e00ac330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 --2- 172.21.15.97:0/264932474 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f93e00a8a30 0x7f93e01d7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 >> 172.21.15.97:0/264932474 conn(0x7f93e0061a30 msgr2=0x7f93e01424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 shutdown_connections 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 -- 172.21.15.97:0/264932474 wait complete. 2023-12-07T07:33:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.076+0000 7f93e556c640 1 librados: shutdown 2023-12-07T07:33:47.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.078+0000 7fada1ffb640 1 -- 172.21.15.97:0/4189246836 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fad9c01e070 con 0x7fada40c5760 2023-12-07T07:33:47.080 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.078+0000 7fada1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:47.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.078+0000 7fada1ffb640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:47.081 INFO:teuthology.orchestra.run.smithi097.stdout:73014444036 2023-12-07T07:33:47.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 10 librados: watch_flush enter 2023-12-07T07:33:47.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 10 librados: watch_flush exit 2023-12-07T07:33:47.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 20 client.4360.objecter shutdown clearing up homeless session... 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 10 client.4360.objecter successfully canceled tick 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fad9006c820 msgr2=0x7fad9006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fad9006c820 0x7fad9006ece0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fada8036000 tx=0x7fada8030000 comp rx=0 tx=0).stop 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 10 monclient: shutdown 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c5760 msgr2=0x7fada4032980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.080+0000 7fadb24ff640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c5760 0x7fada4032980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00b4a0 tx=0x7fad9c00b960 comp rx=0 tx=0).stop 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 shutdown_connections 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fad9006c820 0x7fad9006ece0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fada402e890 0x7fada402ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c5760 0x7fada4032980 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 --2- 172.21.15.97:0/4189246836 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fada4038d70 0x7fada4032280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 >> 172.21.15.97:0/4189246836 conn(0x7fada40065d0 msgr2=0x7fada40cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:47.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 shutdown_connections 2023-12-07T07:33:47.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 -- 172.21.15.97:0/4189246836 wait complete. 2023-12-07T07:33:47.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.081+0000 7fadb24ff640 1 librados: shutdown 2023-12-07T07:33:47.091 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.6 2023-12-07T07:33:47.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444036 for osd.3 2023-12-07T07:33:47.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-07T07:33:47.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.477+0000 7efe8dbf0640 10 monclient: get_monmap_and_config 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.477+0000 7efe8dbf0640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.477+0000 7efe8dbf0640 10 monclient: monmap: 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:47.478620+0000 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:47.478620+0000 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:47.480 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:47.481 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:47.481 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 1 Processor -- start 2023-12-07T07:33:47.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 1 -- start start 2023-12-07T07:33:47.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient: init 2023-12-07T07:33:47.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.483 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efe880a8950 0x7efe8813f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient(hunting): picked mon.noname-a con 0x7efe880a8950 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe880a8d30 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe88145ec0 0x7efe8813f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient(hunting): picked mon.noname-c con 0x7efe88145ec0 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe88145320 con 0x7efe880a8d30 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe880b2260 con 0x7efe880a8950 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe880aa5e0 con 0x7efe88145ec0 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.478+0000 7efe8dbf0640 10 monclient(hunting): authenticate will time out at 1349.857209s 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe87fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe88145ec0 0x7efe8813f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe877fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7efe880a8950 0x7efe8813f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe86ffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efe880a8d30 0x7efe881457b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41300/0 (socket says 172.21.15.97:41300) 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe86ffd640 1 -- 172.21.15.97:0/3016823709 learned_addr learned my addr 172.21.15.97:0/3016823709 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe87fff640 10 monclient(hunting): get_auth_request con 0x7efe88145ec0 auth_method 0 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe87fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.479+0000 7efe87fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.480+0000 7efe877fe640 10 monclient(hunting): get_auth_request con 0x7efe880a8950 auth_method 0 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.480+0000 7efe877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.480+0000 7efe877fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.480+0000 7efe877fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.480+0000 7efe87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.480+0000 7efe87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.480+0000 7efe87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.481+0000 7efe877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.481+0000 7efe877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.481+0000 7efe877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2023-12-07T07:33:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 1 -- 172.21.15.97:0/3016823709 >> v2:172.21.15.97:3300/0 conn(0x7efe880a8950 msgr2=0x7efe8813f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 1 --2- 172.21.15.97:0/3016823709 >> v2:172.21.15.97:3300/0 conn(0x7efe880a8950 0x7efe8813f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 1 -- 172.21.15.97:0/3016823709 >> v1:172.21.15.97:6789/0 conn(0x7efe880a8d30 legacy=0x7efe881457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 1 monclient: found mon.noname-c 2023-12-07T07:33:47.487 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.482+0000 7efe87fff640 1 -- 172.21.15.97:0/3016823709 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe880aa8c0 con 0x7efe88145ec0 2023-12-07T07:33:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.484+0000 7efe87fff640 1 --2- 172.21.15.97:0/3016823709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe88145ec0 0x7efe8813f6d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efe880b8550 tx=0x7efe8002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c84cdc07bad2ce9d server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.489 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 1 -- 172.21.15.97:0/3016823709 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8002cc80 con 0x7efe88145ec0 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: dump: 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.490 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 1 -- 172.21.15.97:0/3016823709 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8002cde0 con 0x7efe88145ec0 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 1 -- 172.21.15.97:0/3016823709 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe80034660 con 0x7efe88145ec0 2023-12-07T07:33:47.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:47.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe867fc640 10 monclient: dump: 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe8dbf0640 5 monclient: authenticate success, global_id 4363 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe8dbf0640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:47.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.487+0000 7efe8dbf0640 4 set_mon_vals no callback set 2023-12-07T07:33:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 10 monclient: shutdown 2023-12-07T07:33:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 -- 172.21.15.97:0/3016823709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe88145ec0 msgr2=0x7efe8813f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/3016823709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe88145ec0 0x7efe8813f6d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efe880b8550 tx=0x7efe8002bed0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 -- 172.21.15.97:0/3016823709 shutdown_connections 2023-12-07T07:33:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/3016823709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe88145ec0 0x7efe8813f6d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/3016823709 >> v2:172.21.15.97:3300/0 conn(0x7efe880a8950 0x7efe8813f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 -- 172.21.15.97:0/3016823709 >> 172.21.15.97:0/3016823709 conn(0x7efe88061970 msgr2=0x7efe880b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 -- 172.21.15.97:0/3016823709 shutdown_connections 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.488+0000 7efe8dbf0640 1 -- 172.21.15.97:0/3016823709 wait complete. 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 10 monclient: monmap: 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.499 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 1 librados: starting msgr at 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 1 librados: starting objecter 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 1 Processor -- start 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 1 -- start start 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 1 librados: setting wanted keys 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 1 librados: calling monclient init 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.489+0000 7efe8dbf0640 10 monclient: init 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe880a8950 0x7efe88137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): picked mon.noname-b con 0x7efe880a8950 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:47.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe880a8d30 0x7efe8813fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): picked mon.noname-a con 0x7efe880a8d30 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7efe88145ec0 0x7efe88142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): picked mon.noname-c con 0x7efe88145ec0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe88142d50 con 0x7efe880a8d30 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe881da5e0 con 0x7efe880a8950 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe881da270 con 0x7efe88145ec0 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe8dbf0640 10 monclient(hunting): authenticate will time out at 1349.868430s 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe86ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe880a8d30 0x7efe8813fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe87fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7efe88145ec0 0x7efe88142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe877fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe880a8950 0x7efe88137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe86ffd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe880a8d30 0x7efe8813fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39046/0 (socket says 172.21.15.97:39046) 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe86ffd640 1 -- 172.21.15.97:0/127081207 learned_addr learned my addr 172.21.15.97:0/127081207 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe86ffd640 10 monclient(hunting): get_auth_request con 0x7efe880a8d30 auth_method 0 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe86ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe86ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe87fff640 10 monclient(hunting): get_auth_request con 0x7efe88145ec0 auth_method 0 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe87fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.490+0000 7efe87fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe877fe640 10 monclient(hunting): get_auth_request con 0x7efe880a8950 auth_method 0 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe877fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe877fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.491+0000 7efe877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 1 -- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7efe88145ec0 msgr2=0x7efe88142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7efe88145ec0 0x7efe88142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 1 -- 172.21.15.97:0/127081207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe880a8950 msgr2=0x7efe88137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe880a8950 0x7efe88137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 1 monclient: found mon.noname-a 2023-12-07T07:33:47.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.494+0000 7efe86ffd640 1 -- 172.21.15.97:0/127081207 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe80009a70 con 0x7efe880a8d30 2023-12-07T07:33:47.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.496+0000 7efe86ffd640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe880a8d30 0x7efe8813fc20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00c500 tx=0x7efe7c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.496+0000 7efe6ffff640 10 client.?.objecter ms_handle_connect 0x7efe880a8d30 2023-12-07T07:33:47.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.496+0000 7efe6ffff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:47.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 1 -- 172.21.15.97:0/127081207 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c005ce0 con 0x7efe880a8d30 2023-12-07T07:33:47.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:47.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: dump: 2023-12-07T07:33:47.506 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.506 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.506 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 1 -- 172.21.15.97:0/127081207 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c005e40 con 0x7efe880a8d30 2023-12-07T07:33:47.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 1 -- 172.21.15.97:0/127081207 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0072f0 con 0x7efe880a8d30 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe6ffff640 10 monclient: dump: 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe8dbf0640 5 monclient: authenticate success, global_id 4329 2023-12-07T07:33:47.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe8dbf0640 10 monclient: _renew_subs 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe8dbf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe88141f90 con 0x7efe880a8d30 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.499+0000 7efe85ffb640 4 set_mon_vals no callback set 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 10 monclient: _renew_subs 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe881dacb0 con 0x7efe880a8d30 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 1 librados: init done 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.500+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe88075b60 con 0x7efe880a8d30 2023-12-07T07:33:47.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.519+0000 7efe6ffff640 1 -- 172.21.15.97:0/127081207 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efe7c007510 con 0x7efe880a8d30 2023-12-07T07:33:47.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.519+0000 7efe6ffff640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7efe7406c7d0 0x7efe7406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.519+0000 7efe877fe640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7efe7406c7d0 0x7efe7406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.519+0000 7efe877fe640 10 monclient: get_auth_request con 0x7efe7406c7d0 auth_method 0 2023-12-07T07:33:47.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.519+0000 7efe877fe640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7efe7406c7d0 0x7efe7406ec90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efe78009b40 tx=0x7efe78009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.519+0000 7efe6ffff640 10 client.4329.objecter ms_handle_connect 0x7efe7406c7d0 2023-12-07T07:33:47.525 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.522+0000 7efe6ffff640 1 -- 172.21.15.97:0/127081207 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efe7c058130 con 0x7efe880a8d30 2023-12-07T07:33:47.525 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.522+0000 7efe6ffff640 10 client.4329.objecter ms_dispatch 0x7efe88000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:47.525 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.522+0000 7efe6ffff640 3 client.4329.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:47.525 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.522+0000 7efe6ffff640 3 client.4329.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:47.525 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.522+0000 7efe6ffff640 20 client.4329.objecter dump_active .. 0 homeless 2023-12-07T07:33:47.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-07T07:33:47.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.660+0000 7f9529ec9640 10 monclient: get_monmap_and_config 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.660+0000 7f9529ec9640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.661+0000 7f9529ec9640 10 monclient: monmap: 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:47.662062+0000 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:47.662062+0000 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.663 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:47.664 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:47.664 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.661+0000 7f9529ec9640 1 Processor -- start 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.661+0000 7f9529ec9640 1 -- start start 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.661+0000 7f9529ec9640 10 monclient: init 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95240a8a30 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9524142bf0 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f95240a8e10 0x7f952413f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f95240a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240aac10 con 0x7f95240a8a30 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240aaef0 con 0x7f9524142bf0 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95240aaa10 con 0x7f95240a8e10 2023-12-07T07:33:47.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9529ec9640 10 monclient(hunting): authenticate will time out at 1350.040709s 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.662+0000 7f9523fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f95240a8e10 0x7f952413f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9522ffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f9524142bf0 0x7f952413ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44750/0 (socket says 172.21.15.97:44750) 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9522ffd640 1 -- 172.21.15.97:0/3749332703 learned_addr learned my addr 172.21.15.97:0/3749332703 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9523fff640 10 monclient(hunting): get_auth_request con 0x7f95240a8e10 auth_method 0 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9523fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9523fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.663+0000 7f9523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 1 -- 172.21.15.97:0/3749332703 >> v1:172.21.15.154:6789/0 conn(0x7f9524142bf0 legacy=0x7f952413ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 1 -- 172.21.15.97:0/3749332703 >> v1:172.21.15.97:6789/0 conn(0x7f95240a8a30 legacy=0x7f95241424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 1 monclient: found mon.noname-b 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f95227fc640 1 -- 172.21.15.97:0/3749332703 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028965503 0 0) 0x7f95240aaef0 con 0x7f9524142bf0 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f9523fff640 1 -- 172.21.15.97:0/3749332703 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95240a97f0 con 0x7f95240a8e10 2023-12-07T07:33:47.668 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.665+0000 7f95227fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-07T07:33:47.670 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.667+0000 7f9523fff640 1 --2- 172.21.15.97:0/3749332703 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f95240a8e10 0x7f952413f400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f951c0097d0 tx=0x7f951c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90765ded544a0ee1 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 1 -- 172.21.15.97:0/3749332703 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c031800 con 0x7f95240a8e10 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: dump: 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.674 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 1 -- 172.21.15.97:0/3749332703 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c031960 con 0x7f95240a8e10 2023-12-07T07:33:47.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f9529ec9640 5 monclient: authenticate success, global_id 4319 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f9529ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 1 -- 172.21.15.97:0/3749332703 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c031cf0 con 0x7f95240a8e10 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f9529ec9640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f9529ec9640 4 set_mon_vals no callback set 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.671+0000 7f95227fc640 10 monclient: dump: 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.676 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.677 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.677 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.672+0000 7f9529ec9640 10 monclient: shutdown 2023-12-07T07:33:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.672+0000 7f9529ec9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.672+0000 7f9529ec9640 1 -- 172.21.15.97:0/3749332703 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f95240a8e10 msgr2=0x7f952413f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.672+0000 7f9529ec9640 1 --2- 172.21.15.97:0/3749332703 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f95240a8e10 0x7f952413f400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f951c0097d0 tx=0x7f951c02b8a0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.672+0000 7f9529ec9640 1 -- 172.21.15.97:0/3749332703 shutdown_connections 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.672+0000 7f9529ec9640 1 --2- 172.21.15.97:0/3749332703 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f95240a8e10 0x7f952413f400 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.672+0000 7f9529ec9640 1 -- 172.21.15.97:0/3749332703 >> 172.21.15.97:0/3749332703 conn(0x7f9524061a30 msgr2=0x7f95240b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.673+0000 7f9529ec9640 1 -- 172.21.15.97:0/3749332703 shutdown_connections 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.673+0000 7f9529ec9640 1 -- 172.21.15.97:0/3749332703 wait complete. 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.673+0000 7f9529ec9640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.673+0000 7f9529ec9640 10 monclient: monmap: 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.674+0000 7f9529ec9640 1 librados: starting msgr at 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.674+0000 7f9529ec9640 1 librados: starting objecter 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.674+0000 7f9529ec9640 1 Processor -- start 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.674+0000 7f9529ec9640 1 -- start start 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.674+0000 7f9529ec9640 1 librados: setting wanted keys 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.674+0000 7f9529ec9640 1 librados: calling monclient init 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.674+0000 7f9529ec9640 10 monclient: init 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95240a8a30 0x7f9524131980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f95240a8a30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95240a8e10 0x7f95241e3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f95240a8e10 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9524142bf0 0x7f95241e4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f9524142bf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95241e4840 con 0x7f95240a8e10 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9524133280 con 0x7f95240a8a30 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9524134790 con 0x7f9524142bf0 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.675+0000 7f9529ec9640 10 monclient(hunting): authenticate will time out at 1350.054022s 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.676+0000 7f9523fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9524142bf0 0x7f95241e4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f95237fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95240a8a30 0x7f9524131980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9524142bf0 0x7f95241e4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38280/0 (socket says 172.21.15.97:38280) 2023-12-07T07:33:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 1 -- 172.21.15.97:0/2098101550 learned_addr learned my addr 172.21.15.97:0/2098101550 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 10 monclient(hunting): get_auth_request con 0x7f9524142bf0 auth_method 0 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.677+0000 7f9523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.679+0000 7f95237fe640 10 monclient(hunting): get_auth_request con 0x7f95240a8a30 auth_method 0 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.679+0000 7f95237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.679+0000 7f95237fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.685 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.679+0000 7f95237fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.680+0000 7f95237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.680+0000 7f95237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.680+0000 7f95237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 1 -- 172.21.15.97:0/2098101550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95240a8a30 msgr2=0x7f9524131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95240a8a30 0x7f9524131980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 1 -- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95240a8e10 msgr2=0x7f95241e3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95240a8e10 0x7f95241e3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 1 monclient: found mon.noname-c 2023-12-07T07:33:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.682+0000 7f9523fff640 1 -- 172.21.15.97:0/2098101550 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c009400 con 0x7f9524142bf0 2023-12-07T07:33:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.685+0000 7f9523fff640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9524142bf0 0x7f95241e4300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f951c02beb0 tx=0x7f951c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.685+0000 7f951bfff640 10 client.?.objecter ms_handle_connect 0x7f9524142bf0 2023-12-07T07:33:47.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.685+0000 7f951bfff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 1 -- 172.21.15.97:0/2098101550 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c03bc40 con 0x7f9524142bf0 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 10 monclient: dump: 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.693 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f951bfff640 1 -- 172.21.15.97:0/2098101550 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c03bda0 con 0x7f9524142bf0 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f9529ec9640 5 monclient: authenticate success, global_id 4322 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f9529ec9640 10 monclient: _renew_subs 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f9529ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.689+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95241362e0 con 0x7f9524142bf0 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f951bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f951bfff640 1 -- 172.21.15.97:0/2098101550 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c045610 con 0x7f9524142bf0 2023-12-07T07:33:47.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f951bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f951bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f951bfff640 10 monclient: dump: 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 10 monclient: _renew_subs 2023-12-07T07:33:47.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:47.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9524131ec0 con 0x7f9524142bf0 2023-12-07T07:33:47.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 1 librados: init done 2023-12-07T07:33:47.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9521ffb640 4 set_mon_vals no callback set 2023-12-07T07:33:47.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:47.696 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.690+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9524067ac0 con 0x7f9524142bf0 2023-12-07T07:33:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.732+0000 7f3d3153e640 10 monclient: get_monmap_and_config 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.732+0000 7f3d3153e640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.732+0000 7f3d3153e640 10 monclient: monmap: 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:47.733757+0000 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:47.733757+0000 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:47.735 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 1 Processor -- start 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 1 -- start start 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient: init 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d2c0a8a30 0x7f3d2c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2c0a8a30 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d2c0a8e10 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d2c142a20 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c142e70 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c1462c0 con 0x7f3d2c142a20 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c0aaa10 con 0x7f3d2c0a8a30 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.733+0000 7f3d3153e640 10 monclient(hunting): authenticate will time out at 1350.112198s 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.734+0000 7f3d2affd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d2c0a8a30 0x7f3d2c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d2c0a8a30 0x7f3d2c1424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38756/0 (socket says 172.21.15.97:38756) 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 1 -- 172.21.15.97:0/1548768369 learned_addr learned my addr 172.21.15.97:0/1548768369 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 10 monclient(hunting): get_auth_request con 0x7f3d2c0a8a30 auth_method 0 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.738 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.735+0000 7f3d2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-12-07T07:33:47.739 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 1 -- 172.21.15.97:0/1548768369 >> v1:172.21.15.154:6789/0 conn(0x7f3d2c142a20 legacy=0x7f3d2c145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 1 -- 172.21.15.97:0/1548768369 >> v1:172.21.15.97:6789/0 conn(0x7f3d2c0a8e10 legacy=0x7f3d2c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 1 monclient: found mon.noname-c 2023-12-07T07:33:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.737+0000 7f3d2affd640 1 -- 172.21.15.97:0/1548768369 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c0aac10 con 0x7f3d2c0a8a30 2023-12-07T07:33:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.740+0000 7f3d2affd640 1 --2- 172.21.15.97:0/1548768369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8a30 0x7f3d2c1424e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009870 tx=0x7f3d1c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3181623984dd90 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 1 -- 172.21.15.97:0/1548768369 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c02f840 con 0x7f3d2c0a8a30 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: dump: 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d3153e640 5 monclient: authenticate success, global_id 4375 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 1 -- 172.21.15.97:0/1548768369 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c02f9a0 con 0x7f3d2c0a8a30 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d3153e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d3153e640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 1 -- 172.21.15.97:0/1548768369 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c02fd30 con 0x7f3d2c0a8a30 2023-12-07T07:33:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d3153e640 4 set_mon_vals no callback set 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.744+0000 7f3d29ffb640 10 monclient: dump: 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 10 monclient: shutdown 2023-12-07T07:33:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 1 -- 172.21.15.97:0/1548768369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8a30 msgr2=0x7f3d2c1424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1548768369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8a30 0x7f3d2c1424e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009870 tx=0x7f3d1c02ba20 comp rx=0 tx=0).stop 2023-12-07T07:33:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 1 -- 172.21.15.97:0/1548768369 shutdown_connections 2023-12-07T07:33:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1548768369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8a30 0x7f3d2c1424e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 1 -- 172.21.15.97:0/1548768369 >> 172.21.15.97:0/1548768369 conn(0x7f3d2c061a30 msgr2=0x7f3d2c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 1 -- 172.21.15.97:0/1548768369 shutdown_connections 2023-12-07T07:33:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 1 -- 172.21.15.97:0/1548768369 wait complete. 2023-12-07T07:33:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.745+0000 7f3d3153e640 10 monclient: monmap: 2023-12-07T07:33:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 librados: starting msgr at 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 librados: starting objecter 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 Processor -- start 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 -- start start 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 librados: setting wanted keys 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 librados: calling monclient init 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient: init 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3d2c0a8a30 0x7f3d2c0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2c0a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8e10 0x7f3d2c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): picked mon.noname-b con 0x7f3d2c0a8e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d2c142a20 0x7f3d2c1da9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): picked mon.noname-a con 0x7f3d2c142a20 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c1da520 con 0x7f3d2c142a20 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0ac170 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0adf80 con 0x7f3d2c0a8a30 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.746+0000 7f3d3153e640 10 monclient(hunting): authenticate will time out at 1350.125259s 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.747+0000 7f3d2a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8e10 0x7f3d2c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.747+0000 7f3d2a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8e10 0x7f3d2c0aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38768/0 (socket says 172.21.15.97:38768) 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.747+0000 7f3d2a7fc640 1 -- 172.21.15.97:0/1600260569 learned_addr learned my addr 172.21.15.97:0/1600260569 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.747+0000 7f3d2a7fc640 10 monclient(hunting): get_auth_request con 0x7f3d2c0a8e10 auth_method 0 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.747+0000 7f3d2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.747+0000 7f3d2a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.747+0000 7f3d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.748+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.748+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.748+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.750+0000 7f3d2a7fc640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.750+0000 7f3d2a7fc640 1 -- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3d2c0a8a30 msgr2=0x7f3d2c0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.750+0000 7f3d2a7fc640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3d2c0a8a30 0x7f3d2c0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.751+0000 7f3d2a7fc640 1 -- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d2c142a20 msgr2=0x7f3d2c1da9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.751+0000 7f3d2a7fc640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d2c142a20 0x7f3d2c1da9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.751+0000 7f3d2a7fc640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.751+0000 7f3d2a7fc640 1 monclient: found mon.noname-b 2023-12-07T07:33:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.751+0000 7f3d2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.751+0000 7f3d2a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:47.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.751+0000 7f3d2a7fc640 1 -- 172.21.15.97:0/1600260569 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0094d0 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.753+0000 7f3d2a7fc640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8e10 0x7f3d2c0aedc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000c410 tx=0x7f3d2000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.753+0000 7f3d137fe640 10 client.?.objecter ms_handle_connect 0x7f3d2c0a8e10 2023-12-07T07:33:47.757 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.753+0000 7f3d137fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 1 -- 172.21.15.97:0/1600260569 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d200035f0 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: dump: 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.759 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 1 -- 172.21.15.97:0/1600260569 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d20003750 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 1 -- 172.21.15.97:0/1600260569 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20003b40 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 5 monclient: authenticate success, global_id 4378 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 10 monclient: _renew_subs 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c0af300 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:47.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d297fa640 4 set_mon_vals no callback set 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d137fe640 10 monclient: dump: 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 10 monclient: _renew_subs 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:47.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c1db0d0 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.756+0000 7f3d3153e640 1 librados: init done 2023-12-07T07:33:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.757+0000 7f3d3153e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.757+0000 7f3d3153e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.757+0000 7f3d3153e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.757+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c067ac0 con 0x7f3d2c0a8e10 2023-12-07T07:33:47.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.774+0000 7efe6ffff640 1 -- 172.21.15.97:0/127081207 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe7c054470 con 0x7efe880a8d30 2023-12-07T07:33:47.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.774+0000 7efe6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.774+0000 7efe6ffff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:47.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.913+0000 7fa1a52cf640 10 monclient: get_monmap_and_config 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.913+0000 7fa1a52cf640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.913+0000 7fa1a52cf640 10 monclient: monmap: 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:47.914189+0000 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:47.914189+0000 2023-12-07T07:33:47.915 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:47.916 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:47.916 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.913+0000 7fa1a52cf640 1 Processor -- start 2023-12-07T07:33:47.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.913+0000 7fa1a52cf640 1 -- start start 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.913+0000 7fa1a52cf640 10 monclient: init 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1a00a8910 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1a00a9b70 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa1a013f370 0x7fa1a01426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a013f370 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00a8cf0 con 0x7fa1a00a9b70 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00aabd0 con 0x7fa1a00a8910 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a00aaeb0 con 0x7fa1a013f370 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa1a52cf640 10 monclient(hunting): authenticate will time out at 1350.292696s 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.914+0000 7fa19f7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa1a013f370 0x7fa1a01426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa1a013f370 0x7fa1a01426a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39072/0 (socket says 172.21.15.97:39072) 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 1 -- 172.21.15.97:0/1635219546 learned_addr learned my addr 172.21.15.97:0/1635219546 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 10 monclient(hunting): get_auth_request con 0x7fa1a013f370 auth_method 0 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.915+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 1 -- 172.21.15.97:0/1635219546 >> v1:172.21.15.154:6789/0 conn(0x7fa1a00a8910 legacy=0x7fa1a013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 1 -- 172.21.15.97:0/1635219546 >> v1:172.21.15.97:6789/0 conn(0x7fa1a00a9b70 legacy=0x7fa1a0141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 1 monclient: found mon.noname-a 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.918+0000 7fa19f7fe640 1 -- 172.21.15.97:0/1635219546 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a00a9720 con 0x7fa1a013f370 2023-12-07T07:33:47.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.920+0000 7fa19f7fe640 1 --2- 172.21.15.97:0/1635219546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a013f370 0x7fa1a01426a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa198009e50 tx=0x7fa19802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb47428e19cedcd server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 1 -- 172.21.15.97:0/1635219546 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19802ca80 con 0x7fa1a013f370 2023-12-07T07:33:47.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:47.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: dump: 2023-12-07T07:33:47.932 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.932 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.932 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.933 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.933 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 1 -- 172.21.15.97:0/1635219546 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19802cbe0 con 0x7fa1a013f370 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa1a52cf640 5 monclient: authenticate success, global_id 4332 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa1a52cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa1a52cf640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa1a52cf640 4 set_mon_vals no callback set 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 1 -- 172.21.15.97:0/1635219546 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980343f0 con 0x7fa1a013f370 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.925+0000 7fa19dffb640 10 monclient: dump: 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.936 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 10 monclient: shutdown 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 1 -- 172.21.15.97:0/1635219546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a013f370 msgr2=0x7fa1a01426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/1635219546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a013f370 0x7fa1a01426a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa198009e50 tx=0x7fa19802bee0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 1 -- 172.21.15.97:0/1635219546 shutdown_connections 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/1635219546 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a013f370 0x7fa1a01426a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 1 -- 172.21.15.97:0/1635219546 >> 172.21.15.97:0/1635219546 conn(0x7fa1a00b7780 msgr2=0x7fa1a00b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 1 -- 172.21.15.97:0/1635219546 shutdown_connections 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.926+0000 7fa1a52cf640 1 -- 172.21.15.97:0/1635219546 wait complete. 2023-12-07T07:33:47.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 10 monclient: build_initial_monmap 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 10 monclient: monmap: 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 1 librados: starting msgr at 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 1 librados: starting objecter 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 1 Processor -- start 2023-12-07T07:33:47.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 1 -- start start 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 1 librados: setting wanted keys 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 1 librados: calling monclient init 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.927+0000 7fa1a52cf640 10 monclient: init 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1a00a8910 0x7fa1a0137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a00a8910 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa1a013f370 0x7fa1a013a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a013f370 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a00b0fb0 0x7fa1a01327e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a00b0fb0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a013ac60 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a013a8f0 con 0x7fa1a00a8910 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a0132d20 con 0x7fa1a013f370 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa1a52cf640 10 monclient(hunting): authenticate will time out at 1350.306609s 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.928+0000 7fa19f7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a00b0fb0 0x7fa1a01327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a00b0fb0 0x7fa1a01327e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39080/0 (socket says 172.21.15.97:39080) 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 1 -- 172.21.15.97:0/3014498644 learned_addr learned my addr 172.21.15.97:0/3014498644 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 10 monclient(hunting): get_auth_request con 0x7fa1a00b0fb0 auth_method 0 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:47.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:47.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:47.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:47.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:47.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.929+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 1 -- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa1a013f370 msgr2=0x7fa1a013a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa1a013f370 0x7fa1a013a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 1 -- 172.21.15.97:0/3014498644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1a00a8910 msgr2=0x7fa1a0137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1a00a8910 0x7fa1a0137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 1 monclient: found mon.noname-a 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.932+0000 7fa19f7fe640 1 -- 172.21.15.97:0/3014498644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa198009b30 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.934+0000 7fa19f7fe640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a00b0fb0 0x7fa1a01327e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa198036040 tx=0x7fa198004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.934+0000 7fa1877fe640 10 client.?.objecter ms_handle_connect 0x7fa1a00b0fb0 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.934+0000 7fa1877fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 1 -- 172.21.15.97:0/3014498644 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19803e450 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: dump: 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.946 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 5 monclient: authenticate success, global_id 4335 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 1 -- 172.21.15.97:0/3014498644 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19803e5d0 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 1 -- 172.21.15.97:0/3014498644 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19803e980 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 10 monclient: _renew_subs 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a0132f40 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:47.947 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa19d7fa640 4 set_mon_vals no callback set 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1877fe640 10 monclient: dump: 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 10 monclient: _renew_subs 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a0137dd0 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.938+0000 7fa1a52cf640 1 librados: init done 2023-12-07T07:33:47.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.939+0000 7fa1a52cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.939+0000 7fa1a52cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.939+0000 7fa1a52cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.939+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a013ada0 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.958+0000 7fa1877fe640 1 -- 172.21.15.97:0/3014498644 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa19802cd50 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.959+0000 7fa1877fe640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa18c06c820 0x7fa18c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.959+0000 7fa19effd640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa18c06c820 0x7fa18c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.959+0000 7fa19effd640 10 monclient: get_auth_request con 0x7fa18c06c820 auth_method 0 2023-12-07T07:33:47.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.959+0000 7fa19effd640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa18c06c820 0x7fa18c06ece0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa190001560 tx=0x7fa190007990 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.959+0000 7fa1877fe640 10 client.4335.objecter ms_handle_connect 0x7fa18c06c820 2023-12-07T07:33:47.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.962+0000 7fa1877fe640 1 -- 172.21.15.97:0/3014498644 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa19807bdb0 con 0x7fa1a00b0fb0 2023-12-07T07:33:47.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.962+0000 7fa1877fe640 10 client.4335.objecter ms_dispatch 0x7fa1a0000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:47.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.962+0000 7fa1877fe640 3 client.4335.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:47.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.962+0000 7fa1877fe640 3 client.4335.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:47.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.962+0000 7fa1877fe640 20 client.4335.objecter dump_active .. 0 homeless 2023-12-07T07:33:47.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.964+0000 7f951bfff640 1 -- 172.21.15.97:0/2098101550 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f951c045770 con 0x7f9524142bf0 2023-12-07T07:33:47.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.964+0000 7f951bfff640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f950406c850 0x7f950406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:47.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.964+0000 7f95237fe640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f950406c850 0x7f950406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:47.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.964+0000 7f95237fe640 10 monclient: get_auth_request con 0x7f950406c850 auth_method 0 2023-12-07T07:33:47.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.965+0000 7f951bfff640 10 client.4322.objecter ms_handle_connect 0x7f950406c850 2023-12-07T07:33:47.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.965+0000 7f95237fe640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f950406c850 0x7f950406ed10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9514001ef0 tx=0x7f9514009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:47.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.970+0000 7efe8dbf0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T07:33:47.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.970+0000 7efe8dbf0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T07:33:47.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.970+0000 7efe8dbf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:47.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.970+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7efe881da750 con 0x7efe880a8d30 2023-12-07T07:33:47.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 1 -- 172.21.15.97:0/2098101550 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f951c02e070 con 0x7f9524142bf0 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 10 client.4322.objecter ms_dispatch 0x7f9524000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 3 client.4322.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 3 client.4322.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 20 client.4322.objecter dump_active .. 0 homeless 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 1 -- 172.21.15.97:0/2098101550 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f951c077630 con 0x7f9524142bf0 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.980+0000 7f951bfff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:47.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-07T07:33:48.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.998+0000 7f3d137fe640 1 -- 172.21.15.97:0/1600260569 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3d20003ca0 con 0x7f3d2c0a8e10 2023-12-07T07:33:48.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.998+0000 7f3d137fe640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f3d1806c820 0x7f3d1806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.999+0000 7f3d2affd640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f3d1806c820 0x7f3d1806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.999+0000 7f3d2affd640 10 monclient: get_auth_request con 0x7f3d1806c820 auth_method 0 2023-12-07T07:33:48.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.999+0000 7f3d137fe640 10 client.4378.objecter ms_handle_connect 0x7f3d1806c820 2023-12-07T07:33:48.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:47.999+0000 7f3d2affd640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f3d1806c820 0x7f3d1806ece0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c036000 tx=0x7f3d1c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.000+0000 7f3d137fe640 1 -- 172.21.15.97:0/1600260569 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3d20058610 con 0x7f3d2c0a8e10 2023-12-07T07:33:48.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.000+0000 7f3d137fe640 10 client.4378.objecter ms_dispatch 0x7f3d2c000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:48.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.000+0000 7f3d137fe640 3 client.4378.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:48.003 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.000+0000 7f3d137fe640 3 client.4378.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:48.003 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.000+0000 7f3d137fe640 20 client.4378.objecter dump_active .. 0 homeless 2023-12-07T07:33:48.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.011+0000 7f3d137fe640 1 -- 172.21.15.97:0/1600260569 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d20054250 con 0x7f3d2c0a8e10 2023-12-07T07:33:48.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.011+0000 7f3d137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.011+0000 7f3d137fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:48.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.079+0000 7f6235409640 10 monclient: get_monmap_and_config 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.079+0000 7f6235409640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.079+0000 7f6235409640 10 monclient: monmap: 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:48.080502+0000 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:48.080502+0000 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:48.082 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:48.083 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:48.083 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.080+0000 7f6235409640 1 Processor -- start 2023-12-07T07:33:48.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.080+0000 7f6235409640 1 -- start start 2023-12-07T07:33:48.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.080+0000 7f6235409640 10 monclient: init 2023-12-07T07:33:48.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62300a8a30 0x7f62301424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): picked mon.noname-b con 0x7f62300a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62300a8e10 0x7f6230142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): picked mon.noname-c con 0x7f62300a8e10 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f62301454a0 0x7f6230145930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): picked mon.noname-a con 0x7f62301454a0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62300aaa10 con 0x7f62301454a0 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62300aac10 con 0x7f62300a8e10 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62300aaef0 con 0x7f62300a8a30 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.081+0000 7f6235409640 10 monclient(hunting): authenticate will time out at 1350.459915s 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f622effd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62300a8a30 0x7f62301424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f6227fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62300a8e10 0x7f6230142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f6227fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62300a8e10 0x7f6230142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38790/0 (socket says 172.21.15.97:38790) 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f622effd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62300a8a30 0x7f62301424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38310/0 (socket says 172.21.15.97:38310) 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f6227fff640 1 -- 172.21.15.97:0/1517938598 learned_addr learned my addr 172.21.15.97:0/1517938598 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f6227fff640 10 monclient(hunting): get_auth_request con 0x7f62300a8e10 auth_method 0 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f6227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f6227fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f6227fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f622effd640 10 monclient(hunting): get_auth_request con 0x7f62300a8a30 auth_method 0 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f622effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f622effd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.082+0000 7f622effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.083+0000 7f6227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.083+0000 7f6227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.083+0000 7f6227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.083+0000 7f622effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.083+0000 7f622effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.083+0000 7f622effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 1 -- 172.21.15.97:0/1517938598 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62300a8a30 msgr2=0x7f62301424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 1 --2- 172.21.15.97:0/1517938598 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62300a8a30 0x7f62301424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 1 -- 172.21.15.97:0/1517938598 >> v2:172.21.15.97:3300/0 conn(0x7f62301454a0 msgr2=0x7f6230145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 1 --2- 172.21.15.97:0/1517938598 >> v2:172.21.15.97:3300/0 conn(0x7f62301454a0 0x7f6230145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 1 monclient: found mon.noname-c 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.084+0000 7f6227fff640 1 -- 172.21.15.97:0/1517938598 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62300a97f0 con 0x7f62300a8e10 2023-12-07T07:33:48.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.086+0000 7f6227fff640 1 --2- 172.21.15.97:0/1517938598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8e10 0x7f6230142a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6218009870 tx=0x7f621802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7bab4df668a14e1 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.088+0000 7f1b73ac2640 10 monclient: get_monmap_and_config 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.088+0000 7f1b73ac2640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.088+0000 7f1b73ac2640 10 monclient: monmap: 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:48.089982+0000 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:48.089982+0000 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:48.091 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:48.092 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-07T07:33:48.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.089+0000 7f1b73ac2640 1 Processor -- start 2023-12-07T07:33:48.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.089+0000 7f1b73ac2640 1 -- start start 2023-12-07T07:33:48.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.089+0000 7f1b73ac2640 10 monclient: init 2023-12-07T07:33:48.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-12-07T07:33:48.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b6c0a8a30 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:48.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1b6c142bf0 0x7f1b6c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient(hunting): picked mon.noname-a con 0x7f1b6c142bf0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b6c0a8e10 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0aac10 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0aaef0 con 0x7f1b6c0a8a30 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c0aaa10 con 0x7f1b6c142bf0 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f1b73ac2640 10 monclient(hunting): authenticate will time out at 1350.468704s 2023-12-07T07:33:48.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 1 -- 172.21.15.97:0/1517938598 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6218031870 con 0x7f62300a8e10 2023-12-07T07:33:48.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:48.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: dump: 2023-12-07T07:33:48.095 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.095 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f6235409640 5 monclient: authenticate success, global_id 4381 2023-12-07T07:33:48.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f6235409640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 1 -- 172.21.15.97:0/1517938598 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62180319d0 con 0x7f62300a8e10 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 1 -- 172.21.15.97:0/1517938598 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6218031da0 con 0x7f62300a8e10 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f622e7fc640 10 monclient: dump: 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.097 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f6235409640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:48.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.090+0000 7f6235409640 4 set_mon_vals no callback set 2023-12-07T07:33:48.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f1b7185e640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1b6c0a8a30 0x7f1b6c1424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44774/0 (socket says 172.21.15.97:44774) 2023-12-07T07:33:48.098 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f1b7185e640 1 -- 172.21.15.97:0/3346477007 learned_addr learned my addr 172.21.15.97:0/3346477007 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:48.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 10 monclient: shutdown 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 -- 172.21.15.97:0/1517938598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8e10 msgr2=0x7f6230142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 --2- 172.21.15.97:0/1517938598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8e10 0x7f6230142a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6218009870 tx=0x7f621802b910 comp rx=0 tx=0).stop 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 -- 172.21.15.97:0/1517938598 shutdown_connections 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 --2- 172.21.15.97:0/1517938598 >> v2:172.21.15.97:3300/0 conn(0x7f62301454a0 0x7f6230145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 --2- 172.21.15.97:0/1517938598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8e10 0x7f6230142a20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 --2- 172.21.15.97:0/1517938598 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62300a8a30 0x7f62301424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 -- 172.21.15.97:0/1517938598 >> 172.21.15.97:0/1517938598 conn(0x7f6230061a30 msgr2=0x7f62300b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 -- 172.21.15.97:0/1517938598 shutdown_connections 2023-12-07T07:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.091+0000 7f6235409640 1 -- 172.21.15.97:0/1517938598 wait complete. 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 10 monclient: monmap: 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 1 librados: starting msgr at 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 1 librados: starting objecter 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 1 Processor -- start 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 1 -- start start 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 1 librados: setting wanted keys 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 1 librados: calling monclient init 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.092+0000 7f6235409640 10 monclient: init 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8a30 0x7f62300adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): picked mon.noname-b con 0x7f62300a8a30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62300a8e10 0x7f62300aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): picked mon.noname-a con 0x7f62300a8e10 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62301454a0 0x7f62301dab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): picked mon.noname-c con 0x7f62301454a0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:48.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62301d4c80 con 0x7f62300a8e10 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62300af130 con 0x7f62300a8a30 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62300ad0d0 con 0x7f62301454a0 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f6235409640 10 monclient(hunting): authenticate will time out at 1350.471379s 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.093+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62301454a0 0x7f62301dab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62301454a0 0x7f62301dab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38322/0 (socket says 172.21.15.97:38322) 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 1 -- 172.21.15.97:0/3513370038 learned_addr learned my addr 172.21.15.97:0/3513370038 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 10 monclient(hunting): get_auth_request con 0x7f62301454a0 auth_method 0 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622effd640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8a30 0x7f62300adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622effd640 10 monclient(hunting): get_auth_request con 0x7f62300a8a30 auth_method 0 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622effd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.094+0000 7f622effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.095+0000 7f622effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.095+0000 7f622effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.095+0000 7f622effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 1 -- 172.21.15.97:0/3513370038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8a30 msgr2=0x7f62300adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8a30 0x7f62300adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 1 -- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62300a8e10 msgr2=0x7f62300aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62300a8e10 0x7f62300aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 1 monclient: found mon.noname-c 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.097+0000 7f622f7fe640 1 -- 172.21.15.97:0/3513370038 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180094d0 con 0x7f62301454a0 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.100+0000 7f622f7fe640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62301454a0 0x7f62301dab80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f622800b9a0 tx=0x7f622800be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.100+0000 7f62277fe640 10 client.?.objecter ms_handle_connect 0x7f62301454a0 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.100+0000 7f62277fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 1 -- 172.21.15.97:0/3513370038 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622800cba0 con 0x7f62301454a0 2023-12-07T07:33:48.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: dump: 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.112 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 1 -- 172.21.15.97:0/3513370038 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622800cd20 con 0x7f62301454a0 2023-12-07T07:33:48.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 1 -- 172.21.15.97:0/3513370038 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228014680 con 0x7f62301454a0 2023-12-07T07:33:48.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:48.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f62277fe640 10 monclient: dump: 2023-12-07T07:33:48.114 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.114 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.114 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.115 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.115 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.115 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.115 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.115 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.115 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.115 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.104+0000 7f622dffb640 4 set_mon_vals no callback set 2023-12-07T07:33:48.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 5 monclient: authenticate success, global_id 4328 2023-12-07T07:33:48.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 10 monclient: _renew_subs 2023-12-07T07:33:48.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623013b700 con 0x7f62301454a0 2023-12-07T07:33:48.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:48.116 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 10 monclient: _renew_subs 2023-12-07T07:33:48.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62301da500 con 0x7f62301454a0 2023-12-07T07:33:48.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.105+0000 7f6235409640 1 librados: init done 2023-12-07T07:33:48.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.106+0000 7f6235409640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.117 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.106+0000 7f6235409640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.107+0000 7f6235409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.107+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6230067ac0 con 0x7f62301454a0 2023-12-07T07:33:48.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.209+0000 7f55744a9640 10 monclient: get_monmap_and_config 2023-12-07T07:33:48.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.209+0000 7f55744a9640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.209+0000 7f55744a9640 10 monclient: monmap: 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:48.210681+0000 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:48.210681+0000 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.210+0000 7f55744a9640 1 Processor -- start 2023-12-07T07:33:48.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.210+0000 7f55744a9640 1 -- start start 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.210+0000 7f55744a9640 10 monclient: init 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f556c0a8a30 0x7f556c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient(hunting): picked mon.noname-c con 0x7f556c0a8a30 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f556c0a8e10 0x7f556c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient(hunting): picked mon.noname-a con 0x7f556c0a8e10 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f556c1454a0 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0aac10 con 0x7f556c1454a0 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0aaa10 con 0x7f556c0a8e10 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0aaef0 con 0x7f556c0a8a30 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f55744a9640 10 monclient(hunting): authenticate will time out at 1350.589769s 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.211+0000 7f5572245640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f556c0a8a30 0x7f556c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f556c0a8a30 0x7f556c1424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38808/0 (socket says 172.21.15.97:38808) 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 1 -- 172.21.15.97:0/368923514 learned_addr learned my addr 172.21.15.97:0/368923514 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 10 monclient(hunting): get_auth_request con 0x7f556c0a8a30 auth_method 0 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.212+0000 7f5572245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2023-12-07T07:33:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 1 -- 172.21.15.97:0/368923514 >> v2:172.21.15.97:3300/0 conn(0x7f556c0a8e10 msgr2=0x7f556c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 1 --2- 172.21.15.97:0/368923514 >> v2:172.21.15.97:3300/0 conn(0x7f556c0a8e10 0x7f556c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 1 -- 172.21.15.97:0/368923514 >> v1:172.21.15.97:6789/0 conn(0x7f556c1454a0 legacy=0x7f556c1458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 1 monclient: found mon.noname-c 2023-12-07T07:33:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.214+0000 7f5572245640 1 -- 172.21.15.97:0/368923514 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c0a97f0 con 0x7f556c0a8a30 2023-12-07T07:33:48.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.216+0000 7f5572245640 1 --2- 172.21.15.97:0/368923514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8a30 0x7f556c1424e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f556c0b5470 tx=0x7f556802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5e42033506f77bf server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 1 -- 172.21.15.97:0/368923514 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568030860 con 0x7f556c0a8a30 2023-12-07T07:33:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: dump: 2023-12-07T07:33:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.224 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 1 -- 172.21.15.97:0/368923514 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55680309c0 con 0x7f556c0a8a30 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 1 -- 172.21.15.97:0/368923514 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568030d50 con 0x7f556c0a8a30 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.220+0000 7f5571243640 10 monclient: dump: 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.221+0000 7f55744a9640 5 monclient: authenticate success, global_id 4387 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.221+0000 7f55744a9640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.221+0000 7f55744a9640 4 set_mon_vals no callback set 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.221+0000 7f55744a9640 10 monclient: shutdown 2023-12-07T07:33:48.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 1 -- 172.21.15.97:0/368923514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8a30 msgr2=0x7f556c1424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 1 --2- 172.21.15.97:0/368923514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8a30 0x7f556c1424e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f556c0b5470 tx=0x7f556802b9a0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 1 -- 172.21.15.97:0/368923514 shutdown_connections 2023-12-07T07:33:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 1 --2- 172.21.15.97:0/368923514 >> v2:172.21.15.97:3300/0 conn(0x7f556c0a8e10 0x7f556c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 1 --2- 172.21.15.97:0/368923514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8a30 0x7f556c1424e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 1 -- 172.21.15.97:0/368923514 >> 172.21.15.97:0/368923514 conn(0x7f556c061a30 msgr2=0x7f556c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.222+0000 7f55744a9640 1 -- 172.21.15.97:0/368923514 shutdown_connections 2023-12-07T07:33:48.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.227+0000 7f55744a9640 1 -- 172.21.15.97:0/368923514 wait complete. 2023-12-07T07:33:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.228+0000 7f55744a9640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.228+0000 7f55744a9640 10 monclient: monmap: 2023-12-07T07:33:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:48.231 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.229+0000 7f55744a9640 1 librados: starting msgr at 2023-12-07T07:33:48.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.229+0000 7f55744a9640 1 librados: starting objecter 2023-12-07T07:33:48.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.229+0000 7f55744a9640 1 Processor -- start 2023-12-07T07:33:48.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.229+0000 7f55744a9640 1 -- start start 2023-12-07T07:33:48.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.230+0000 7f55744a9640 1 librados: setting wanted keys 2023-12-07T07:33:48.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.230+0000 7f55744a9640 1 librados: calling monclient init 2023-12-07T07:33:48.232 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.230+0000 7f55744a9640 10 monclient: init 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.231+0000 7f55744a9640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.231+0000 7f55744a9640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.231+0000 7f55744a9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c0a8a30 0x7f556c1da930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.231+0000 7f55744a9640 10 monclient(hunting): picked mon.noname-a con 0x7f556c0a8a30 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.231+0000 7f55744a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8e10 0x7f556c0af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.231+0000 7f55744a9640 10 monclient(hunting): picked mon.noname-b con 0x7f556c0a8e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.231+0000 7f55744a9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f556c1454a0 0x7f556c0af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 10 monclient(hunting): picked mon.noname-c con 0x7f556c1454a0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0aef40 con 0x7f556c0a8a30 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0adf70 con 0x7f556c0a8e10 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f5571a44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8e10 0x7f556c0af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0aead0 con 0x7f556c1454a0 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f5571a44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8e10 0x7f556c0af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38820/0 (socket says 172.21.15.97:38820) 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f5571a44640 1 -- 172.21.15.97:0/2994618470 learned_addr learned my addr 172.21.15.97:0/2994618470 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f55744a9640 10 monclient(hunting): authenticate will time out at 1350.611032s 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f5571a44640 10 monclient(hunting): get_auth_request con 0x7f556c0a8e10 auth_method 0 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f5571a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f5571a44640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.236 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.232+0000 7f5571a44640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.233+0000 7f5571a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.233+0000 7f5571a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.237 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.233+0000 7f5571a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 1 -- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f556c1454a0 msgr2=0x7f556c0af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f556c1454a0 0x7f556c0af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 1 -- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c0a8a30 msgr2=0x7f556c1da930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c0a8a30 0x7f556c1da930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 1 monclient: found mon.noname-b 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.236+0000 7f5571a44640 1 -- 172.21.15.97:0/2994618470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55680094d0 con 0x7f556c0a8e10 2023-12-07T07:33:48.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.239+0000 7f5571a44640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8e10 0x7f556c0af3d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b4a0 tx=0x7f555c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.239+0000 7f5562ffd640 10 client.?.objecter ms_handle_connect 0x7f556c0a8e10 2023-12-07T07:33:48.241 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.239+0000 7f5562ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:48.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 1 -- 172.21.15.97:0/2994618470 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c010610 con 0x7f556c0a8e10 2023-12-07T07:33:48.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:48.244 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: dump: 2023-12-07T07:33:48.244 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 1 -- 172.21.15.97:0/2994618470 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c010770 con 0x7f556c0a8e10 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.245 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 1 -- 172.21.15.97:0/2994618470 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c010b20 con 0x7f556c0a8e10 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5562ffd640 10 monclient: dump: 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.246 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.242+0000 7f5570a42640 4 set_mon_vals no callback set 2023-12-07T07:33:48.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 5 monclient: authenticate success, global_id 4390 2023-12-07T07:33:48.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 10 monclient: _renew_subs 2023-12-07T07:33:48.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c0aff30 con 0x7f556c0a8e10 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 10 monclient: _renew_subs 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c0ac3a0 con 0x7f556c0a8e10 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.243+0000 7f55744a9640 1 librados: init done 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.244+0000 7f55744a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.244+0000 7f55744a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.244+0000 7f55744a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.244+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c067ac0 con 0x7f556c0a8e10 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.246+0000 7f2b2f232640 10 monclient: get_monmap_and_config 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.246+0000 7f2b2f232640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.247+0000 7f2b2f232640 10 monclient: monmap: 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:48.248066+0000 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:48.248066+0000 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:48.250 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.247+0000 7f2b2f232640 1 Processor -- start 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.247+0000 7f2b2f232640 1 -- start start 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.247+0000 7f2b2f232640 10 monclient: init 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b280a8a30 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b28142bf0 0x7f2b2813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 10 monclient(hunting): picked mon.noname-c con 0x7f2b28142bf0 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b280a8e10 0x7f2b2813f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.251 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 10 monclient(hunting): picked mon.noname-b con 0x7f2b280a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:48.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.248+0000 7f2b2f232640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280aac10 con 0x7f2b280a8a30 2023-12-07T07:33:48.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.249+0000 7f2b2f232640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.249+0000 7f2b2f232640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b280aaa10 con 0x7f2b28142bf0 2023-12-07T07:33:48.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.249+0000 7f2b2f232640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.249+0000 7f2b2f232640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b280aaef0 con 0x7f2b280a8e10 2023-12-07T07:33:48.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.249+0000 7f2b2f232640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.249+0000 7f2b2f232640 10 monclient(hunting): authenticate will time out at 1350.627316s 2023-12-07T07:33:48.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.290+0000 7f1b7105d640 1 --2- 172.21.15.97:0/3346477007 >> v2:172.21.15.97:3300/0 conn(0x7f1b6c142bf0 0x7f1b6c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.291+0000 7f1b7105d640 10 monclient(hunting): get_auth_request con 0x7f1b6c142bf0 auth_method 0 2023-12-07T07:33:48.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.291+0000 7f1b7105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.291+0000 7f1b7105d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.291+0000 7f1b7105d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.292+0000 7f1b7105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.292+0000 7f1b7105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.292+0000 7f1b7105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 1 -- 172.21.15.97:0/3346477007 >> v1:172.21.15.154:6789/0 conn(0x7f1b6c0a8a30 legacy=0x7f1b6c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 1 -- 172.21.15.97:0/3346477007 >> v1:172.21.15.97:6789/0 conn(0x7f1b6c0a8e10 legacy=0x7f1b6c13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 1 monclient: found mon.noname-a 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:48.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.297+0000 7f1b7105d640 1 -- 172.21.15.97:0/3346477007 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c0a97f0 con 0x7f1b6c142bf0 2023-12-07T07:33:48.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.312+0000 7fa1877fe640 1 -- 172.21.15.97:0/3014498644 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1980779f0 con 0x7fa1a00b0fb0 2023-12-07T07:33:48.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.312+0000 7fa1877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.312+0000 7fa1877fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:48.324 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.322+0000 7f1b7105d640 1 --2- 172.21.15.97:0/3346477007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c142bf0 0x7f1b6c13ecf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0097a0 tx=0x7f1b5c02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42dcea605ff73fe server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.325+0000 7f2b2cfce640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2b280a8a30 0x7f2b281424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41370/0 (socket says 172.21.15.97:41370) 2023-12-07T07:33:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.325+0000 7f2b2cfce640 1 -- 172.21.15.97:0/3051909964 learned_addr learned my addr 172.21.15.97:0/3051909964 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.333+0000 7efe6ffff640 1 -- 172.21.15.97:0/127081207 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efe7c021090 con 0x7efe880a8d30 2023-12-07T07:33:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.333+0000 7efe6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-07T07:33:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.333+0000 7efe6ffff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:48.335 INFO:teuthology.orchestra.run.smithi097.stdout:73014444037 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 10 librados: watch_flush enter 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 10 librados: watch_flush exit 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 20 client.4329.objecter shutdown clearing up homeless session... 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 10 client.4329.objecter successfully canceled tick 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7efe7406c7d0 msgr2=0x7efe7406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7efe7406c7d0 0x7efe7406ec90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efe78009b40 tx=0x7efe78009190 comp rx=0 tx=0).stop 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 10 monclient: shutdown 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe880a8d30 msgr2=0x7efe8813fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe880a8d30 0x7efe8813fc20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00c500 tx=0x7efe7c00c9c0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 shutdown_connections 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7efe7406c7d0 0x7efe7406ec90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7efe88145ec0 0x7efe88142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efe880a8d30 0x7efe8813fc20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 --2- 172.21.15.97:0/127081207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe880a8950 0x7efe88137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 >> 172.21.15.97:0/127081207 conn(0x7efe88061970 msgr2=0x7efe88137170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 shutdown_connections 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 -- 172.21.15.97:0/127081207 wait complete. 2023-12-07T07:33:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.335+0000 7efe8dbf0640 1 librados: shutdown 2023-12-07T07:33:48.353 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444037 for osd.2 2023-12-07T07:33:48.353 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:48.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.353+0000 7f9529ec9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:48.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.353+0000 7f9529ec9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:48.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.353+0000 7f9529ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.353+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f95241345f0 con 0x7f9524142bf0 2023-12-07T07:33:48.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.377+0000 7f3d3153e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:48.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.377+0000 7f3d3153e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:48.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.377+0000 7f3d3153e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.377+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3d2c1da690 con 0x7f3d2c0a8e10 2023-12-07T07:33:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.422+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546660311 0 0) 0x7f2b280aac10 con 0x7f2b280a8a30 2023-12-07T07:33:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.422+0000 7f2b277fe640 10 monclient(hunting): _negotiate 2023-12-07T07:33:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.422+0000 7f2b277fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.422+0000 7f2b277fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.422+0000 7f2b277fe640 10 monclient(hunting): my global_id is 4341 2023-12-07T07:33:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.422+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b18003520 con 0x7f2b280a8a30 2023-12-07T07:33:48.425 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f62277fe640 1 -- 172.21.15.97:0/3513370038 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f622801d3b0 con 0x7f62301454a0 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 1 -- 172.21.15.97:0/3346477007 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c02f830 con 0x7f1b6c142bf0 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: dump: 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.426 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 1 -- 172.21.15.97:0/3346477007 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c02f990 con 0x7f1b6c142bf0 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 1 -- 172.21.15.97:0/3346477007 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c02fd60 con 0x7f1b6c142bf0 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b7085c640 10 monclient: dump: 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.427 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b73ac2640 5 monclient: authenticate success, global_id 4338 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b73ac2640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f1b73ac2640 4 set_mon_vals no callback set 2023-12-07T07:33:48.428 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f62277fe640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f621006c7d0 0x7f621006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.423+0000 7f622effd640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f621006c7d0 0x7f621006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f622effd640 10 monclient: get_auth_request con 0x7f621006c7d0 auth_method 0 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 10 monclient: shutdown 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 1 -- 172.21.15.97:0/3346477007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c142bf0 msgr2=0x7f1b6c13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/3346477007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c142bf0 0x7f1b6c13ecf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0097a0 tx=0x7f1b5c02ba10 comp rx=0 tx=0).stop 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f622effd640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f621006c7d0 0x7f621006ec90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6220009b40 tx=0x7f6220009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.429 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f62277fe640 10 client.4328.objecter ms_handle_connect 0x7f621006c7d0 2023-12-07T07:33:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 1 -- 172.21.15.97:0/3346477007 shutdown_connections 2023-12-07T07:33:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/3346477007 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c142bf0 0x7f1b6c13ecf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 1 -- 172.21.15.97:0/3346477007 >> 172.21.15.97:0/3346477007 conn(0x7f1b6c061a30 msgr2=0x7f1b6c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 1 -- 172.21.15.97:0/3346477007 shutdown_connections 2023-12-07T07:33:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.424+0000 7f1b73ac2640 1 -- 172.21.15.97:0/3346477007 wait complete. 2023-12-07T07:33:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.425+0000 7f1b73ac2640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.431 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.425+0000 7f1b73ac2640 10 monclient: monmap: 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.425+0000 7f1b73ac2640 1 librados: starting msgr at 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.425+0000 7f1b73ac2640 1 librados: starting objecter 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 Processor -- start 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 -- start start 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 librados: setting wanted keys 2023-12-07T07:33:48.432 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 librados: calling monclient init 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient: init 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b6c0a8a30 0x7f1b6c0b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): picked mon.noname-b con 0x7f1b6c0a8a30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c0a8e10 0x7f1b6c1449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): picked mon.noname-a con 0x7f1b6c0a8e10 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1b6c142bf0 0x7f1b6c144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): picked mon.noname-c con 0x7f1b6c142bf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c143b90 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c143dc0 con 0x7f1b6c0a8a30 2023-12-07T07:33:48.433 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c1440a0 con 0x7f1b6c142bf0 2023-12-07T07:33:48.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.434 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.426+0000 7f1b73ac2640 10 monclient(hunting): authenticate will time out at 1350.805081s 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3005808211 0 0) 0x7f2b18003520 con 0x7f2b280a8a30 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b280a8e10 msgr2=0x7f2b2813f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b280a8e10 0x7f2b2813f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 >> v2:172.21.15.154:3300/0 conn(0x7f2b28142bf0 msgr2=0x7f2b2813ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 --2- 172.21.15.97:0/3051909964 >> v2:172.21.15.154:3300/0 conn(0x7f2b28142bf0 0x7f2b2813ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 monclient: found mon.noname-a-legacy 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.97:6789/0 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b280a97f0 con 0x7f2b280a8a30 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.435 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2176119509 0 0) 0x7f2b1c001670 con 0x7f2b280a8a30 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: dump: 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] but i'm connected to v1:172.21.15.97:6789/0, reconnecting 2023-12-07T07:33:48.436 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 >> v1:172.21.15.97:6789/0 conn(0x7f2b280a8a30 legacy=0x7f2b281424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b280a8e10 0x7f2b18003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): picked mon.c con 0x7f2b280a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b180040d0 0x7f2b180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): picked mon.b con 0x7f2b180040d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 0x7f2b1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): picked mon.a con 0x7f2b18004e20 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b280aaa10 con 0x7f2b18004e20 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b280aaef0 con 0x7f2b180040d0 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180055a0 con 0x7f2b280a8e10 2023-12-07T07:33:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.428+0000 7f2b277fe640 10 monclient(hunting): _finish_auth 0 2023-12-07T07:33:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f1b7105d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c0a8e10 0x7f1b6c1449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f2b2d7cf640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 0x7f2b1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f1b7105d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c0a8e10 0x7f1b6c1449a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39154/0 (socket says 172.21.15.97:39154) 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f1b7105d640 1 -- 172.21.15.97:0/6056570 learned_addr learned my addr 172.21.15.97:0/6056570 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f1b7105d640 10 monclient(hunting): get_auth_request con 0x7f1b6c0a8e10 auth_method 0 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f1b7105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f1b7105d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.433+0000 7f1b7105d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.434+0000 7f2b2d7cf640 10 monclient(hunting): get_auth_request con 0x7f2b18004e20 auth_method 0 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.434+0000 7f2b2d7cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.434+0000 7f2b2d7cf640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.434+0000 7f2b2d7cf640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:33:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.435+0000 7f1b7105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.435+0000 7f1b7105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.435+0000 7f1b7105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.435+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.435+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.440 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.435+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:33:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2023-12-07T07:33:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 1 -- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1b6c142bf0 msgr2=0x7f1b6c144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1b6c142bf0 0x7f1b6c144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 1 -- 172.21.15.97:0/6056570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b6c0a8a30 msgr2=0x7f1b6c0b7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b6c0a8a30 0x7f1b6c0b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 1 monclient: found mon.noname-a 2023-12-07T07:33:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:48.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.439+0000 7f1b7105d640 1 -- 172.21.15.97:0/6056570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c009400 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_done global_id 4341 payload 293 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b280a8e10 msgr2=0x7f2b18003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b280a8e10 0x7f2b18003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/3051909964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b180040d0 msgr2=0x7f2b180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b180040d0 0x7f2b180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 1 monclient: found mon.a 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/3051909964 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18003520 con 0x7f2b18004e20 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2d7cf640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.444 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2f232640 5 monclient: authenticate success, global_id 4341 2023-12-07T07:33:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.441+0000 7f2b2f232640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 1 -- 172.21.15.97:0/3513370038 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f62280587b0 con 0x7f62301454a0 2023-12-07T07:33:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 10 client.4328.objecter ms_dispatch 0x7f6230000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 3 client.4328.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 3 client.4328.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:48.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 20 client.4328.objecter dump_active .. 0 homeless 2023-12-07T07:33:48.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 1 -- 172.21.15.97:0/3513370038 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6228054af0 con 0x7f62301454a0 2023-12-07T07:33:48.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.442+0000 7f62277fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:48.448 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.446+0000 7f2b2d7cf640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 0x7f2b1800b980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b20009e40 tx=0x7f2b180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53eab6eedd768ac3 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.447+0000 7f1b7105d640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c0a8e10 0x7f1b6c1449a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0098c0 tx=0x7f1b5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.447+0000 7f1b627fc640 10 client.?.objecter ms_handle_connect 0x7f1b6c0a8e10 2023-12-07T07:33:48.449 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.447+0000 7f1b627fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:48.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2002cd90 con 0x7f2b18004e20 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 10 monclient: dump: 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.456 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b20034440 con 0x7f2b18004e20 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b2f232640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b2f232640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 1 -- 172.21.15.97:0/3051909964 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b200347f0 con 0x7f2b18004e20 2023-12-07T07:33:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b2f232640 4 set_mon_vals no callback set 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.453+0000 7f2b277fe640 10 monclient: dump: 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.458 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.459 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.459 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 1 -- 172.21.15.97:0/6056570 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c03b040 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: dump: 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 1 -- 172.21.15.97:0/6056570 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c02f7d0 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b63fff640 4 set_mon_vals no callback set 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2f232640 10 monclient: shutdown 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2f232640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2f232640 1 -- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 msgr2=0x7f2b1800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2f232640 1 --2- 172.21.15.97:0/3051909964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 0x7f2b1800b980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b20009e40 tx=0x7f2b180057f0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/3051909964 reap_dead start 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2f232640 1 -- 172.21.15.97:0/3051909964 shutdown_connections 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2f232640 1 -- 172.21.15.97:0/3051909964 >> 172.21.15.97:0/3051909964 conn(0x7f2b28061a30 msgr2=0x7f2b280b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.462 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f2b2f232640 1 -- 172.21.15.97:0/3051909964 shutdown_connections 2023-12-07T07:33:48.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 1 -- 172.21.15.97:0/6056570 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0426c0 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:48.463 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b627fc640 10 monclient: dump: 2023-12-07T07:33:48.463 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.463 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.463 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.464 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.464 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 5 monclient: authenticate success, global_id 4344 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 10 monclient: _renew_subs 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c0b59d0 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 10 monclient: _renew_subs 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c0b7880 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.465 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.454+0000 7f1b73ac2640 1 librados: init done 2023-12-07T07:33:48.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.455+0000 7f2b2f232640 1 -- 172.21.15.97:0/3051909964 wait complete. 2023-12-07T07:33:48.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f1b73ac2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f1b73ac2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.466 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f1b73ac2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:48.467 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 10 monclient: build_initial_monmap 2023-12-07T07:33:48.467 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 10 monclient: monmap: 2023-12-07T07:33:48.467 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:48.467 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:48.467 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:48.467 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:48.467 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:48.468 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.468 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:48.468 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:48.468 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:48.468 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.468 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c067ac0 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 1 librados: starting msgr at 2023-12-07T07:33:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 1 librados: starting objecter 2023-12-07T07:33:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 1 Processor -- start 2023-12-07T07:33:48.472 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 1 -- start start 2023-12-07T07:33:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 1 librados: setting wanted keys 2023-12-07T07:33:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 1 librados: calling monclient init 2023-12-07T07:33:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.456+0000 7f2b2f232640 10 monclient: init 2023-12-07T07:33:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:33:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 0x7f2b281441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.473 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): picked mon.noname-a con 0x7f2b18004e20 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b28144d70 0x7f2b281d7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): picked mon.noname-b con 0x7f2b28144d70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b281d7780 0x7f2b280b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): picked mon.noname-c con 0x7f2b281d7780 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b281d4b00 con 0x7f2b18004e20 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b28137010 con 0x7f2b28144d70 2023-12-07T07:33:48.474 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b281451b0 con 0x7f2b281d7780 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.457+0000 7f2b2f232640 10 monclient(hunting): authenticate will time out at 1350.835842s 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.465+0000 7f2b2d7cf640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b281d7780 0x7f2b280b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.466+0000 7f2b2d7cf640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b281d7780 0x7f2b280b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38368/0 (socket says 172.21.15.97:38368) 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.466+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/190728601 learned_addr learned my addr 172.21.15.97:0/190728601 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.466+0000 7f2b2d7cf640 10 monclient(hunting): get_auth_request con 0x7f2b281d7780 auth_method 0 2023-12-07T07:33:48.475 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.466+0000 7f2b2d7cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.466+0000 7f2b2d7cf640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.466+0000 7f2b2d7cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.467+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.467+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:48.476 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.467+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:48.477 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/190728601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b28144d70 msgr2=0x7f2b281d7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b28144d70 0x7f2b281d7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 msgr2=0x7f2b281441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 0x7f2b281441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 10 monclient: _finish_hunting 0 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 1 monclient: found mon.noname-c 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.471+0000 7f2b2d7cf640 1 -- 172.21.15.97:0/190728601 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b28143390 con 0x7f2b281d7780 2023-12-07T07:33:48.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.473+0000 7f2b2d7cf640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b281d7780 0x7f2b280b0c30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002c800 tx=0x7f2b20004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.473+0000 7f2b257fa640 10 client.?.objecter ms_handle_connect 0x7f2b281d7780 2023-12-07T07:33:48.479 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.473+0000 7f2b257fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:48.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.492+0000 7f5562ffd640 1 -- 172.21.15.97:0/2994618470 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f555c010c80 con 0x7f556c0a8e10 2023-12-07T07:33:48.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.493+0000 7f5562ffd640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f555006c7d0 0x7f555006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.493+0000 7f5572245640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f555006c7d0 0x7f555006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.493+0000 7f5572245640 10 monclient: get_auth_request con 0x7f555006c7d0 auth_method 0 2023-12-07T07:33:48.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.493+0000 7f5562ffd640 10 client.4390.objecter ms_handle_connect 0x7f555006c7d0 2023-12-07T07:33:48.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.493+0000 7f5572245640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f555006c7d0 0x7f555006ec90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5568036000 tx=0x7f556803b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.497+0000 7f5562ffd640 1 -- 172.21.15.97:0/2994618470 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f555c058b00 con 0x7f556c0a8e10 2023-12-07T07:33:48.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.497+0000 7f5562ffd640 10 client.4390.objecter ms_dispatch 0x7f556c000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:48.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.497+0000 7f5562ffd640 3 client.4390.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:48.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.497+0000 7f5562ffd640 3 client.4390.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:48.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.497+0000 7f5562ffd640 20 client.4390.objecter dump_active .. 0 homeless 2023-12-07T07:33:48.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.507+0000 7f5562ffd640 1 -- 172.21.15.97:0/2994618470 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f555c054740 con 0x7f556c0a8e10 2023-12-07T07:33:48.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.507+0000 7f5562ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.507+0000 7f5562ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:48.515 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.512+0000 7f3d137fe640 1 -- 172.21.15.97:0/1600260569 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d2c1da690 con 0x7f3d2c0a8e10 2023-12-07T07:33:48.515 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.512+0000 7f3d137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:48.515 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.512+0000 7f3d137fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:48.515 INFO:teuthology.orchestra.run.smithi097.stdout:73014444037 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 10 librados: watch_flush enter 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 10 librados: watch_flush exit 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 20 client.4378.objecter shutdown clearing up homeless session... 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 10 client.4378.objecter successfully canceled tick 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f3d1806c820 msgr2=0x7f3d1806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f3d1806c820 0x7f3d1806ece0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c036000 tx=0x7f3d1c030000 comp rx=0 tx=0).stop 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 10 monclient: shutdown 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8e10 msgr2=0x7f3d2c0aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8e10 0x7f3d2c0aedc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000c410 tx=0x7f3d2000c8d0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 shutdown_connections 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f3d1806c820 0x7f3d1806ece0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3d2c142a20 0x7f3d2c1da9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0a8e10 0x7f3d2c0aedc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 --2- 172.21.15.97:0/1600260569 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3d2c0a8a30 0x7f3d2c0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 >> 172.21.15.97:0/1600260569 conn(0x7f3d2c061a30 msgr2=0x7f3d2c145bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.515+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 shutdown_connections 2023-12-07T07:33:48.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.516+0000 7f3d3153e640 1 -- 172.21.15.97:0/1600260569 wait complete. 2023-12-07T07:33:48.519 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.516+0000 7f3d3153e640 1 librados: shutdown 2023-12-07T07:33:48.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444037 for osd.1 2023-12-07T07:33:48.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.541+0000 7f951bfff640 1 -- 172.21.15.97:0/2098101550 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f95241345f0 con 0x7f9524142bf0 2023-12-07T07:33:48.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.541+0000 7f951bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:48.544 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.541+0000 7f951bfff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:48.544 INFO:teuthology.orchestra.run.smithi097.stdout:55834574853 2023-12-07T07:33:48.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 1 -- 172.21.15.97:0/190728601 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2003e410 con 0x7f2b281d7780 2023-12-07T07:33:48.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.545 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: dump: 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: _finish_auth 0 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: _check_auth_tickets 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:48.546 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 1 -- 172.21.15.97:0/190728601 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2003e590 con 0x7f2b281d7780 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 5 monclient: authenticate success, global_id 4331 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 10 monclient: _renew_subs 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b281366b0 con 0x7f2b281d7780 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 1 -- 172.21.15.97:0/190728601 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2003e940 con 0x7f2b281d7780 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b257fa640 10 monclient: dump: 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:48.547 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 10 monclient: _renew_subs 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b280b1170 con 0x7f2b281d7780 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b2f232640 1 librados: init done 2023-12-07T07:33:48.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.543+0000 7f2b26ffd640 4 set_mon_vals no callback set 2023-12-07T07:33:48.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 10 librados: watch_flush enter 2023-12-07T07:33:48.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 10 librados: watch_flush exit 2023-12-07T07:33:48.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 20 client.4322.objecter shutdown clearing up homeless session... 2023-12-07T07:33:48.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 10 client.4322.objecter successfully canceled tick 2023-12-07T07:33:48.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f950406c850 msgr2=0x7f950406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f950406c850 0x7f950406ed10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9514001ef0 tx=0x7f9514009040 comp rx=0 tx=0).stop 2023-12-07T07:33:48.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 10 monclient: shutdown 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9524142bf0 msgr2=0x7f95241e4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9524142bf0 0x7f95241e4300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f951c02beb0 tx=0x7f951c0046c0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 shutdown_connections 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f950406c850 0x7f950406ed10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9524142bf0 0x7f95241e4300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f95240a8e10 0x7f95241e3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 --2- 172.21.15.97:0/2098101550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95240a8a30 0x7f9524131980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 >> 172.21.15.97:0/2098101550 conn(0x7f9524061a30 msgr2=0x7f952413e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f2b2f232640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.551 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f2b2f232640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.551 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f2b2f232640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.551 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.545+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 shutdown_connections 2023-12-07T07:33:48.551 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.544+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08005040 con 0x7f2b281d7780 2023-12-07T07:33:48.551 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.545+0000 7f9529ec9640 1 -- 172.21.15.97:0/2098101550 wait complete. 2023-12-07T07:33:48.552 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.545+0000 7f9529ec9640 1 librados: shutdown 2023-12-07T07:33:48.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2023-12-07T07:33:48.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.564+0000 7fa1a52cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:48.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.564+0000 7fa1a52cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:48.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.564+0000 7fa1a52cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:48.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.564+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa1a013ada0 con 0x7fa1a00b0fb0 2023-12-07T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.734+0000 7f6235409640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.735+0000 7f6235409640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.735+0000 7f6235409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:48.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.735+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f62300af270 con 0x7f62301454a0 2023-12-07T07:33:48.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.759+0000 7f1b627fc640 1 -- 172.21.15.97:0/6056570 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1b5c042820 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.761 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.759+0000 7f1b627fc640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f1b5006c800 0x7f1b5006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.759+0000 7f1b7185e640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f1b5006c800 0x7f1b5006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.759+0000 7f1b7185e640 10 monclient: get_auth_request con 0x7f1b5006c800 auth_method 0 2023-12-07T07:33:48.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.760+0000 7f1b7185e640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f1b5006c800 0x7f1b5006ecc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1b680015f0 tx=0x7f1b68007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.762 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.760+0000 7f1b627fc640 10 client.4344.objecter ms_handle_connect 0x7f1b5006c800 2023-12-07T07:33:48.775 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 1 -- 172.21.15.97:0/6056570 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1b5c07c7f0 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.775 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 10 client.4344.objecter ms_dispatch 0x7f1b6c000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:48.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 3 client.4344.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:48.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 3 client.4344.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:48.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 20 client.4344.objecter dump_active .. 0 homeless 2023-12-07T07:33:48.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 1 -- 172.21.15.97:0/6056570 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b5c078430 con 0x7f1b6c0a8e10 2023-12-07T07:33:48.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.773+0000 7f1b627fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.781+0000 7fa1877fe640 1 -- 172.21.15.97:0/3014498644 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa1a013ada0 con 0x7fa1a00b0fb0 2023-12-07T07:33:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.781+0000 7fa1877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:48.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.781+0000 7fa1877fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:48.783 INFO:teuthology.orchestra.run.smithi097.stdout:55834574852 2023-12-07T07:33:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.781+0000 7f55744a9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.781+0000 7f55744a9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.781+0000 7f55744a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:48.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.781+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f556c075b60 con 0x7f556c0a8e10 2023-12-07T07:33:48.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 10 librados: watch_flush enter 2023-12-07T07:33:48.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 10 librados: watch_flush exit 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 20 client.4335.objecter shutdown clearing up homeless session... 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 10 client.4335.objecter successfully canceled tick 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa18c06c820 msgr2=0x7fa18c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa18c06c820 0x7fa18c06ece0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa190001560 tx=0x7fa190007990 comp rx=0 tx=0).stop 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 10 monclient: shutdown 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a00b0fb0 msgr2=0x7fa1a01327e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a00b0fb0 0x7fa1a01327e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa198036040 tx=0x7fa198004a40 comp rx=0 tx=0).stop 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 shutdown_connections 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa18c06c820 0x7fa18c06ece0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1a00b0fb0 0x7fa1a01327e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa1a013f370 0x7fa1a013a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 --2- 172.21.15.97:0/3014498644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1a00a8910 0x7fa1a0137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 >> 172.21.15.97:0/3014498644 conn(0x7fa1a00b7780 msgr2=0x7fa1a00b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 shutdown_connections 2023-12-07T07:33:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 -- 172.21.15.97:0/3014498644 wait complete. 2023-12-07T07:33:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.783+0000 7fa1a52cf640 1 librados: shutdown 2023-12-07T07:33:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.785+0000 7f5562ffd640 1 -- 172.21.15.97:0/2994618470 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f555c01e020 con 0x7f556c0a8e10 2023-12-07T07:33:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.785+0000 7f5562ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:48.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.785+0000 7f5562ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:48.787 INFO:teuthology.orchestra.run.smithi097.stdout:73014444036 2023-12-07T07:33:48.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 10 librados: watch_flush enter 2023-12-07T07:33:48.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 10 librados: watch_flush exit 2023-12-07T07:33:48.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 20 client.4390.objecter shutdown clearing up homeless session... 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 10 client.4390.objecter successfully canceled tick 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f555006c7d0 msgr2=0x7f555006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f555006c7d0 0x7f555006ec90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5568036000 tx=0x7f556803b040 comp rx=0 tx=0).stop 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 10 monclient: shutdown 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8e10 msgr2=0x7f556c0af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8e10 0x7f556c0af3d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f555c00b4a0 tx=0x7f555c00b960 comp rx=0 tx=0).stop 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 shutdown_connections 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f555006c7d0 0x7f555006ec90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f556c1454a0 0x7f556c0af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f556c0a8e10 0x7f556c0af3d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 --2- 172.21.15.97:0/2994618470 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f556c0a8a30 0x7f556c1da930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 >> 172.21.15.97:0/2994618470 conn(0x7f556c061a30 msgr2=0x7f556c0b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 shutdown_connections 2023-12-07T07:33:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 -- 172.21.15.97:0/2994618470 wait complete. 2023-12-07T07:33:48.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.788+0000 7f55744a9640 1 librados: shutdown 2023-12-07T07:33:48.801 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2023-12-07T07:33:48.805 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444036 for osd.3 2023-12-07T07:33:48.817 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.815+0000 7f2b257fa640 1 -- 172.21.15.97:0/190728601 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2b2003b050 con 0x7f2b281d7780 2023-12-07T07:33:48.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.816+0000 7f2b257fa640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2b0c06c7d0 0x7f2b0c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:48.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.816+0000 7f2b2cfce640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2b0c06c7d0 0x7f2b0c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:48.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.816+0000 7f2b2cfce640 10 monclient: get_auth_request con 0x7f2b0c06c7d0 auth_method 0 2023-12-07T07:33:48.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.816+0000 7f2b257fa640 10 client.4331.objecter ms_handle_connect 0x7f2b0c06c7d0 2023-12-07T07:33:48.818 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.816+0000 7f2b2cfce640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2b0c06c7d0 0x7f2b0c06ec90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c001eb0 tx=0x7f2b1c0015d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:48.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 1 -- 172.21.15.97:0/190728601 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2b2007b960 con 0x7f2b281d7780 2023-12-07T07:33:48.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 10 client.4331.objecter ms_dispatch 0x7f2b28000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:48.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 3 client.4331.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:48.835 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 3 client.4331.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 20 client.4331.objecter dump_active .. 0 homeless 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 1 -- 172.21.15.97:0/190728601 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b200775a0 con 0x7f2b281d7780 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f2b257fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f62277fe640 1 -- 172.21.15.97:0/3513370038 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6228019070 con 0x7f62301454a0 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f62277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.833+0000 7f62277fe640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:48.836 INFO:teuthology.orchestra.run.smithi097.stdout:68719476741 2023-12-07T07:33:48.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 10 librados: watch_flush enter 2023-12-07T07:33:48.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 10 librados: watch_flush exit 2023-12-07T07:33:48.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 20 client.4328.objecter shutdown clearing up homeless session... 2023-12-07T07:33:48.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 10 client.4328.objecter successfully canceled tick 2023-12-07T07:33:48.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f621006c7d0 msgr2=0x7f621006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f621006c7d0 0x7f621006ec90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6220009b40 tx=0x7f6220009190 comp rx=0 tx=0).stop 2023-12-07T07:33:48.838 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 10 monclient: shutdown 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62301454a0 msgr2=0x7f62301dab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62301454a0 0x7f62301dab80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f622800b9a0 tx=0x7f622800be60 comp rx=0 tx=0).stop 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 shutdown_connections 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f621006c7d0 0x7f621006ec90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f62301454a0 0x7f62301dab80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62300a8e10 0x7f62300aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 --2- 172.21.15.97:0/3513370038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62300a8a30 0x7f62300adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 >> 172.21.15.97:0/3513370038 conn(0x7f6230061a30 msgr2=0x7f623013ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 shutdown_connections 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 -- 172.21.15.97:0/3513370038 wait complete. 2023-12-07T07:33:48.839 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:48.835+0000 7f6235409640 1 librados: shutdown 2023-12-07T07:33:48.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.0 2023-12-07T07:33:48.851 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:49.008 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.006+0000 7f1b73ac2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T07:33:49.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.006+0000 7f1b73ac2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T07:33:49.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.006+0000 7f1b73ac2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:49.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.006+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1b6c0b65a0 con 0x7f1b6c0a8e10 2023-12-07T07:33:49.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.011+0000 7f1b627fc640 1 -- 172.21.15.97:0/6056570 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b6c0b65a0 con 0x7f1b6c0a8e10 2023-12-07T07:33:49.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.011+0000 7f1b627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-07T07:33:49.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.011+0000 7f1b627fc640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:49.014 INFO:teuthology.orchestra.run.smithi097.stdout:55834574854 2023-12-07T07:33:49.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 10 librados: watch_flush enter 2023-12-07T07:33:49.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 10 librados: watch_flush exit 2023-12-07T07:33:49.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 20 client.4344.objecter shutdown clearing up homeless session... 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 10 client.4344.objecter successfully canceled tick 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f1b5006c800 msgr2=0x7f1b5006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f1b5006c800 0x7f1b5006ecc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1b680015f0 tx=0x7f1b68007a40 comp rx=0 tx=0).stop 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 10 monclient: shutdown 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c0a8e10 msgr2=0x7f1b6c1449a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c0a8e10 0x7f1b6c1449a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0098c0 tx=0x7f1b5c0046c0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 shutdown_connections 2023-12-07T07:33:49.017 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f1b5006c800 0x7f1b5006ecc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1b6c142bf0 0x7f1b6c144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1b6c0a8e10 0x7f1b6c1449a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 --2- 172.21.15.97:0/6056570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b6c0a8a30 0x7f1b6c0b7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 >> 172.21.15.97:0/6056570 conn(0x7f1b6c061a30 msgr2=0x7f1b6c13cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 shutdown_connections 2023-12-07T07:33:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 -- 172.21.15.97:0/6056570 wait complete. 2023-12-07T07:33:49.018 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.013+0000 7f1b73ac2640 1 librados: shutdown 2023-12-07T07:33:49.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.5 2023-12-07T07:33:49.027 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:49.052 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.050+0000 7f2b2f232640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T07:33:49.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.050+0000 7f2b2f232640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T07:33:49.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.050+0000 7f2b2f232640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:49.053 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.050+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2b080055c0 con 0x7f2b281d7780 2023-12-07T07:33:49.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.054+0000 7f2b257fa640 1 -- 172.21.15.97:0/190728601 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b20039020 con 0x7f2b281d7780 2023-12-07T07:33:49.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.054+0000 7f2b257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-07T07:33:49.056 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.054+0000 7f2b257fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:49.056 INFO:teuthology.orchestra.run.smithi097.stdout:60129542150 2023-12-07T07:33:49.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 10 librados: watch_flush enter 2023-12-07T07:33:49.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 10 librados: watch_flush exit 2023-12-07T07:33:49.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 20 client.4331.objecter shutdown clearing up homeless session... 2023-12-07T07:33:49.058 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 10 client.4331.objecter successfully canceled tick 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2b0c06c7d0 msgr2=0x7f2b0c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2b0c06c7d0 0x7f2b0c06ec90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c001eb0 tx=0x7f2b1c0015d0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 10 monclient: shutdown 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b281d7780 msgr2=0x7f2b280b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.055+0000 7f2b2f232640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b281d7780 0x7f2b280b0c30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002c800 tx=0x7f2b20004b50 comp rx=0 tx=0).stop 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 shutdown_connections 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2b0c06c7d0 0x7f2b0c06ec90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2b281d7780 0x7f2b280b0c30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b28144d70 0x7f2b281d7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 --2- 172.21.15.97:0/190728601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2b18004e20 0x7f2b281441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 >> 172.21.15.97:0/190728601 conn(0x7f2b28061a30 msgr2=0x7f2b2813af50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 shutdown_connections 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 -- 172.21.15.97:0/190728601 wait complete. 2023-12-07T07:33:49.059 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.056+0000 7f2b2f232640 1 librados: shutdown 2023-12-07T07:33:49.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.6 2023-12-07T07:33:49.069 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:49.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-07T07:33:49.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-07T07:33:49.635 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.633+0000 7fc55f184640 10 monclient: get_monmap_and_config 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.633+0000 7fc55f184640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.633+0000 7fc55f184640 10 monclient: monmap: 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:49.634668+0000 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:49.634668+0000 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:49.636 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.634+0000 7fc55f184640 1 Processor -- start 2023-12-07T07:33:49.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.634+0000 7fc55f184640 1 -- start start 2023-12-07T07:33:49.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.634+0000 7fc55f184640 10 monclient: init 2023-12-07T07:33:49.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.634+0000 7fc55f184640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.634+0000 7fc55f184640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-12-07T07:33:49.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.634+0000 7fc55f184640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5580a8a30 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:49.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc558142bf0 0x7fc55813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 10 monclient(hunting): picked mon.noname-b con 0x7fc558142bf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc5580a8e10 0x7fc55813f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 10 monclient(hunting): picked mon.noname-a con 0x7fc5580a8e10 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580aac10 con 0x7fc5580a8a30 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5580aaa10 con 0x7fc5580a8e10 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580aaef0 con 0x7fc558142bf0 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc55f184640 10 monclient(hunting): authenticate will time out at 1352.013351s 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc557fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc558142bf0 0x7fc55813ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc557fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc558142bf0 0x7fc55813ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38376/0 (socket says 172.21.15.97:38376) 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc557fff640 1 -- 172.21.15.97:0/1149957262 learned_addr learned my addr 172.21.15.97:0/1149957262 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc557fff640 10 monclient(hunting): get_auth_request con 0x7fc558142bf0 auth_method 0 2023-12-07T07:33:49.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc557fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc557fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.635+0000 7fc557fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.636+0000 7fc557fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.636+0000 7fc557fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.636+0000 7fc557fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 1 -- 172.21.15.97:0/1149957262 >> v2:172.21.15.97:3300/0 conn(0x7fc5580a8e10 msgr2=0x7fc55813f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 1 --2- 172.21.15.97:0/1149957262 >> v2:172.21.15.97:3300/0 conn(0x7fc5580a8e10 0x7fc55813f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 1 -- 172.21.15.97:0/1149957262 >> v1:172.21.15.97:6789/0 conn(0x7fc5580a8a30 legacy=0x7fc5581424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 1 monclient: found mon.noname-b 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:49.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.638+0000 7fc557fff640 1 -- 172.21.15.97:0/1149957262 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580a97f0 con 0x7fc558142bf0 2023-12-07T07:33:49.642 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.640+0000 7fc557fff640 1 --2- 172.21.15.97:0/1149957262 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc558142bf0 0x7fc55813ecf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc544009870 tx=0x7fc54402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bb18aedbb6b0564 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 1 -- 172.21.15.97:0/1149957262 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544031640 con 0x7fc558142bf0 2023-12-07T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: dump: 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 1 -- 172.21.15.97:0/1149957262 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5440317a0 con 0x7fc558142bf0 2023-12-07T07:33:49.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc55f184640 5 monclient: authenticate success, global_id 4334 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc55f184640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc55f184640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc55f184640 4 set_mon_vals no callback set 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 1 -- 172.21.15.97:0/1149957262 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544031b30 con 0x7fc558142bf0 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.644+0000 7fc5577fe640 10 monclient: dump: 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.648 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.649 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 10 monclient: shutdown 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 -- 172.21.15.97:0/1149957262 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc558142bf0 msgr2=0x7fc55813ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 --2- 172.21.15.97:0/1149957262 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc558142bf0 0x7fc55813ecf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc544009870 tx=0x7fc54402b8f0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 -- 172.21.15.97:0/1149957262 shutdown_connections 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 --2- 172.21.15.97:0/1149957262 >> v2:172.21.15.97:3300/0 conn(0x7fc5580a8e10 0x7fc55813f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 --2- 172.21.15.97:0/1149957262 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc558142bf0 0x7fc55813ecf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 -- 172.21.15.97:0/1149957262 >> 172.21.15.97:0/1149957262 conn(0x7fc558061a30 msgr2=0x7fc5580b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 -- 172.21.15.97:0/1149957262 shutdown_connections 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.645+0000 7fc55f184640 1 -- 172.21.15.97:0/1149957262 wait complete. 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 10 monclient: monmap: 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:49.656 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 1 librados: starting msgr at 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 1 librados: starting objecter 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 1 Processor -- start 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 1 -- start start 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 1 librados: setting wanted keys 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 1 librados: calling monclient init 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.646+0000 7fc55f184640 10 monclient: init 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc5580a8a30 0x7fc5580adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): picked mon.noname-c con 0x7fc5580a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5580a8e10 0x7fc5580aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): picked mon.noname-b con 0x7fc5580a8e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc558142bf0 0x7fc5581da9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): picked mon.noname-a con 0x7fc558142bf0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:49.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5581da550 con 0x7fc558142bf0 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580aef60 con 0x7fc5580a8e10 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5581d6fd0 con 0x7fc5580a8a30 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc55f184640 10 monclient(hunting): authenticate will time out at 1352.025701s 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.647+0000 7fc557fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5580a8e10 0x7fc5580aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5580a8e10 0x7fc5580aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38850/0 (socket says 172.21.15.97:38850) 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 1 -- 172.21.15.97:0/1949861510 learned_addr learned my addr 172.21.15.97:0/1949861510 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 10 monclient(hunting): get_auth_request con 0x7fc5580a8e10 auth_method 0 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc557fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.648+0000 7fc55cf20640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc5580a8a30 0x7fc5580adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55cf20640 10 monclient(hunting): get_auth_request con 0x7fc5580a8a30 auth_method 0 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55cf20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55cf20640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55cf20640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55d721640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc558142bf0 0x7fc5581da9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55d721640 10 monclient(hunting): get_auth_request con 0x7fc558142bf0 auth_method 0 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55d721640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55d721640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55d721640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55cf20640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55cf20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.649+0000 7fc55cf20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.650+0000 7fc55d721640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.650+0000 7fc55d721640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.650+0000 7fc55d721640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 1 -- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc5580a8a30 msgr2=0x7fc5580adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc5580a8a30 0x7fc5580adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 1 -- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc558142bf0 msgr2=0x7fc5581da9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc558142bf0 0x7fc5581da9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 1 monclient: found mon.noname-b 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.651+0000 7fc557fff640 1 -- 172.21.15.97:0/1949861510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5440094d0 con 0x7fc5580a8e10 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.653+0000 7fc557fff640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5580a8e10 0x7fc5580aea20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5440090b0 tx=0x7fc5440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.653+0000 7fc5557fa640 10 client.?.objecter ms_handle_connect 0x7fc5580a8e10 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.653+0000 7fc5557fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 1 -- 172.21.15.97:0/1949861510 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54403ac60 con 0x7fc5580a8e10 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: dump: 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.662 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 1 -- 172.21.15.97:0/1949861510 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54403adc0 con 0x7fc5580a8e10 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 1 -- 172.21.15.97:0/1949861510 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544044630 con 0x7fc5580a8e10 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 5 monclient: authenticate success, global_id 4393 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 10 monclient: _renew_subs 2023-12-07T07:33:49.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5580b0170 con 0x7fc5580a8e10 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc556ffd640 4 set_mon_vals no callback set 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc5557fa640 10 monclient: dump: 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 10 monclient: _renew_subs 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5581daf20 con 0x7fc5580a8e10 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.657+0000 7fc55f184640 1 librados: init done 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.658+0000 7fc55f184640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.658+0000 7fc55f184640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.658+0000 7fc55f184640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.658+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc558067ac0 con 0x7fc5580a8e10 2023-12-07T07:33:49.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.681+0000 7f07bac75640 10 monclient: get_monmap_and_config 2023-12-07T07:33:49.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.681+0000 7f07bac75640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.683 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.681+0000 7f07bac75640 10 monclient: monmap: 2023-12-07T07:33:49.683 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.683 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.683 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:49.682220+0000 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:49.682220+0000 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:49.684 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.681+0000 7f07bac75640 1 Processor -- start 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.681+0000 7f07bac75640 1 -- start start 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.681+0000 7f07bac75640 10 monclient: init 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07b40a8a30 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f07b4142bf0 0x7f07b413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient(hunting): picked mon.noname-a con 0x7f07b4142bf0 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:49.686 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8e10 0x7f07b413f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient(hunting): picked mon.noname-b con 0x7f07b40a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b40aac10 con 0x7f07b40a8a30 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b40aaa10 con 0x7f07b4142bf0 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40aaef0 con 0x7f07b40a8e10 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.682+0000 7f07bac75640 10 monclient(hunting): authenticate will time out at 1352.060835s 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8e10 0x7f07b413f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f07b4142bf0 0x7f07b413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8e10 0x7f07b413f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38402/0 (socket says 172.21.15.97:38402) 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 1 -- 172.21.15.97:0/2364217014 learned_addr learned my addr 172.21.15.97:0/2364217014 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 10 monclient(hunting): get_auth_request con 0x7f07b40a8e10 auth_method 0 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.687 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 10 monclient(hunting): get_auth_request con 0x7f07b4142bf0 auth_method 0 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b9212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.688 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.683+0000 7f07b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 1 -- 172.21.15.97:0/2364217014 >> v2:172.21.15.97:3300/0 conn(0x7f07b4142bf0 msgr2=0x7f07b413ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 1 --2- 172.21.15.97:0/2364217014 >> v2:172.21.15.97:3300/0 conn(0x7f07b4142bf0 0x7f07b413ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 1 -- 172.21.15.97:0/2364217014 >> v1:172.21.15.154:6789/0 conn(0x7f07b40a8a30 legacy=0x7f07b41424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 1 monclient: found mon.noname-b 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.689 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:33:49.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.685+0000 7f07b9212640 1 -- 172.21.15.97:0/2364217014 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40a97f0 con 0x7f07b40a8e10 2023-12-07T07:33:49.690 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.687+0000 7f07b9212640 1 --2- 172.21.15.97:0/2364217014 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8e10 0x7f07b413f230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009e50 tx=0x7f07ac02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8f05130ef44a6a server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 1 -- 172.21.15.97:0/2364217014 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac02ca70 con 0x7f07b40a8e10 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: dump: 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.693 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 1 -- 172.21.15.97:0/2364217014 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac02cbd0 con 0x7f07b40a8e10 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 5 monclient: authenticate success, global_id 4340 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 1 -- 172.21.15.97:0/2364217014 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0343f0 con 0x7f07b40a8e10 2023-12-07T07:33:49.694 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 4 set_mon_vals no callback set 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07b37fe640 10 monclient: dump: 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.695 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 10 monclient: shutdown 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 -- 172.21.15.97:0/2364217014 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8e10 msgr2=0x7f07b413f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 --2- 172.21.15.97:0/2364217014 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8e10 0x7f07b413f230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009e50 tx=0x7f07ac02bed0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 -- 172.21.15.97:0/2364217014 shutdown_connections 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 --2- 172.21.15.97:0/2364217014 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8e10 0x7f07b413f230 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 --2- 172.21.15.97:0/2364217014 >> v2:172.21.15.97:3300/0 conn(0x7f07b4142bf0 0x7f07b413ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.698 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 -- 172.21.15.97:0/2364217014 >> 172.21.15.97:0/2364217014 conn(0x7f07b4061a30 msgr2=0x7f07b40b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 -- 172.21.15.97:0/2364217014 shutdown_connections 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.691+0000 7f07bac75640 1 -- 172.21.15.97:0/2364217014 wait complete. 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.692+0000 7f07bac75640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.692+0000 7f07bac75640 10 monclient: monmap: 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.692+0000 7f07bac75640 1 librados: starting msgr at 2023-12-07T07:33:49.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.692+0000 7f07bac75640 1 librados: starting objecter 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 Processor -- start 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 -- start start 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 librados: setting wanted keys 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 librados: calling monclient init 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient: init 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8a30 0x7f07b40abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): picked mon.noname-c con 0x7f07b40a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b40a8e10 0x7f07b40ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.700 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): picked mon.noname-b con 0x7f07b40a8e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07b4142bf0 0x7f07b41daa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): picked mon.noname-a con 0x7f07b4142bf0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.701 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40ab580 con 0x7f07b4142bf0 2023-12-07T07:33:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b41da550 con 0x7f07b40a8e10 2023-12-07T07:33:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4145e40 con 0x7f07b40a8a30 2023-12-07T07:33:49.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.693+0000 7f07bac75640 10 monclient(hunting): authenticate will time out at 1352.072166s 2023-12-07T07:33:49.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b8a11640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8a30 0x7f07b40abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b9212640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07b4142bf0 0x7f07b41daa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b8a11640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8a30 0x7f07b40abb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38410/0 (socket says 172.21.15.97:38410) 2023-12-07T07:33:49.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b8a11640 1 -- 172.21.15.97:0/576319231 learned_addr learned my addr 172.21.15.97:0/576319231 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:49.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b8a11640 10 monclient(hunting): get_auth_request con 0x7f07b40a8a30 auth_method 0 2023-12-07T07:33:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b8a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b8a11640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b8a11640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b9212640 10 monclient(hunting): get_auth_request con 0x7f07b4142bf0 auth_method 0 2023-12-07T07:33:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b9212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b9212640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.694+0000 7f07b9212640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.695+0000 7f07b8a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.695+0000 7f07b8a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.695+0000 7f07b8a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.695+0000 7f07b9212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.695+0000 7f07b9212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.695+0000 7f07b9212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2023-12-07T07:33:49.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 1 -- 172.21.15.97:0/576319231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b40a8e10 msgr2=0x7f07b40ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b40a8e10 0x7f07b40ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 1 -- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07b4142bf0 msgr2=0x7f07b41daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07b4142bf0 0x7f07b41daa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 1 monclient: found mon.noname-c 2023-12-07T07:33:49.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:49.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.697+0000 7f07b8a11640 1 -- 172.21.15.97:0/576319231 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac009b30 con 0x7f07b40a8a30 2023-12-07T07:33:49.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.699+0000 7f07b8a11640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8a30 0x7f07b40abb70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f079c00c410 tx=0x7f079c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.699+0000 7f07b17fa640 10 client.?.objecter ms_handle_connect 0x7f07b40a8a30 2023-12-07T07:33:49.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.699+0000 7f07b17fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:49.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 1 -- 172.21.15.97:0/576319231 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c0035f0 con 0x7f07b40a8a30 2023-12-07T07:33:49.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:49.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: dump: 2023-12-07T07:33:49.712 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.712 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.712 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.712 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.712 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.712 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.713 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.713 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.713 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.713 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.713 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 1 -- 172.21.15.97:0/576319231 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c003750 con 0x7f07b40a8a30 2023-12-07T07:33:49.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 1 -- 172.21.15.97:0/576319231 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c003b40 con 0x7f07b40a8a30 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 5 monclient: authenticate success, global_id 4343 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 10 monclient: _renew_subs 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b4144060 con 0x7f07b40a8a30 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b17fa640 10 monclient: dump: 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.715 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07b27fc640 4 set_mon_vals no callback set 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 10 monclient: _renew_subs 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b40acf30 con 0x7f07b40a8a30 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.703+0000 7f07bac75640 1 librados: init done 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.704+0000 7f07bac75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.704+0000 7f07bac75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.704+0000 7f07bac75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.704+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4067ac0 con 0x7f07b40a8a30 2023-12-07T07:33:49.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-07T07:33:49.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-07T07:33:49.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.899+0000 7fc5557fa640 1 -- 172.21.15.97:0/1949861510 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc54402f050 con 0x7fc5580a8e10 2023-12-07T07:33:49.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.899+0000 7fc5557fa640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc53c06c820 0x7fc53c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.899+0000 7fc55cf20640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc53c06c820 0x7fc53c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.899+0000 7fc55cf20640 10 monclient: get_auth_request con 0x7fc53c06c820 auth_method 0 2023-12-07T07:33:49.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.900+0000 7fc5557fa640 10 client.4393.objecter ms_handle_connect 0x7fc53c06c820 2023-12-07T07:33:49.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.900+0000 7fc55cf20640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc53c06c820 0x7fc53c06ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0017f0 tx=0x7fc54c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.901+0000 7fb104f41640 10 monclient: get_monmap_and_config 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.901+0000 7fb104f41640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.901+0000 7fb104f41640 10 monclient: monmap: 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:49.902807+0000 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:49.902807+0000 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:49.904 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fb104f41640 1 Processor -- start 2023-12-07T07:33:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fb104f41640 1 -- start start 2023-12-07T07:33:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fb104f41640 10 monclient: init 2023-12-07T07:33:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fc5557fa640 1 -- 172.21.15.97:0/1949861510 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc54407a550 con 0x7fc5580a8e10 2023-12-07T07:33:49.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fc5557fa640 10 client.4393.objecter ms_dispatch 0x7fc558000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fc5557fa640 3 client.4393.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fc5557fa640 3 client.4393.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:49.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.902+0000 7fc5557fa640 20 client.4393.objecter dump_active .. 0 homeless 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb1000a8a30 0x7fb1001424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient(hunting): picked mon.noname-c con 0x7fb1000a8a30 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1000a8e10 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb100142a20 0x7fb100145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient(hunting): picked mon.noname-b con 0x7fb100142a20 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb100142eb0 con 0x7fb1000a8e10 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1000aaa10 con 0x7fb1000a8a30 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000aac10 con 0x7fb100142a20 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.903+0000 7fb104f41640 10 monclient(hunting): authenticate will time out at 1352.281587s 2023-12-07T07:33:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.912+0000 7fc5557fa640 1 -- 172.21.15.97:0/1949861510 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc544076890 con 0x7fc5580a8e10 2023-12-07T07:33:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.912+0000 7fc5557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.912+0000 7fc5557fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:49.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.916+0000 7fb0fe59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb1000a8a30 0x7fb1001424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.916+0000 7fb0f7fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb1000a8e10 0x7fb1001454a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:44792/0 (socket says 172.21.15.97:44792) 2023-12-07T07:33:49.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.916+0000 7fb0f7fff640 1 -- 172.21.15.97:0/1714266827 learned_addr learned my addr 172.21.15.97:0/1714266827 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:49.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.916+0000 7fb0fe59c640 10 monclient(hunting): get_auth_request con 0x7fb1000a8a30 auth_method 0 2023-12-07T07:33:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.916+0000 7fb0fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.916+0000 7fb0fe59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.916+0000 7fb0fe59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.917+0000 7fb0fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.917+0000 7fb0fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.917+0000 7fb0fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2023-12-07T07:33:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 1 -- 172.21.15.97:0/1714266827 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb100142a20 msgr2=0x7fb100145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 1 --2- 172.21.15.97:0/1714266827 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb100142a20 0x7fb100145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 1 -- 172.21.15.97:0/1714266827 >> v1:172.21.15.154:6789/0 conn(0x7fb1000a8e10 legacy=0x7fb1001454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 1 monclient: found mon.noname-c 2023-12-07T07:33:49.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.919+0000 7fb0fe59c640 1 -- 172.21.15.97:0/1714266827 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1000aaef0 con 0x7fb1000a8a30 2023-12-07T07:33:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.923+0000 7fb0fe59c640 1 --2- 172.21.15.97:0/1714266827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8a30 0x7fb1001424e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097a0 tx=0x7fb0f002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deba71b04a15a379 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 1 -- 172.21.15.97:0/1714266827 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0030610 con 0x7fb1000a8a30 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: dump: 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 1 -- 172.21.15.97:0/1714266827 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f0030770 con 0x7fb1000a8a30 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb104f41640 5 monclient: authenticate success, global_id 4396 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb104f41640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb104f41640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb104f41640 4 set_mon_vals no callback set 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 1 -- 172.21.15.97:0/1714266827 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0030b00 con 0x7fb1000a8a30 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.927+0000 7fb0fdd9b640 10 monclient: dump: 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.931 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.932 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 10 monclient: shutdown 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 -- 172.21.15.97:0/1714266827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8a30 msgr2=0x7fb1001424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 --2- 172.21.15.97:0/1714266827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8a30 0x7fb1001424e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097a0 tx=0x7fb0f002b960 comp rx=0 tx=0).stop 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 -- 172.21.15.97:0/1714266827 shutdown_connections 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 --2- 172.21.15.97:0/1714266827 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb100142a20 0x7fb100145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 --2- 172.21.15.97:0/1714266827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8a30 0x7fb1001424e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 -- 172.21.15.97:0/1714266827 >> 172.21.15.97:0/1714266827 conn(0x7fb100061a30 msgr2=0x7fb1000b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 -- 172.21.15.97:0/1714266827 shutdown_connections 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.928+0000 7fb104f41640 1 -- 172.21.15.97:0/1714266827 wait complete. 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 10 monclient: monmap: 2023-12-07T07:33:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 1 librados: starting msgr at 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 1 librados: starting objecter 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 1 Processor -- start 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 1 -- start start 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 1 librados: setting wanted keys 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 1 librados: calling monclient init 2023-12-07T07:33:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.929+0000 7fb104f41640 10 monclient: init 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb1000a8a30 0x7fb1000afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): picked mon.noname-c con 0x7fb1000a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8e10 0x7fb1000aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): picked mon.noname-b con 0x7fb1000a8e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb100142a20 0x7fb10013b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): picked mon.noname-a con 0x7fb100142a20 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000b1650 con 0x7fb100142a20 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000b0fe0 con 0x7fb1000a8e10 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000adf90 con 0x7fb1000a8a30 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb104f41640 10 monclient(hunting): authenticate will time out at 1352.308696s 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.930+0000 7fb0f7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8e10 0x7fb1000aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0fed9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb100142a20 0x7fb10013b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8e10 0x7fb1000aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:38868/0 (socket says 172.21.15.97:38868) 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 1 -- 172.21.15.97:0/688747607 learned_addr learned my addr 172.21.15.97:0/688747607 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 10 monclient(hunting): get_auth_request con 0x7fb1000a8e10 auth_method 0 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0fed9d640 10 monclient(hunting): get_auth_request con 0x7fb100142a20 auth_method 0 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0fed9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0fed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.931+0000 7fb0f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.932+0000 7fb0fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.932+0000 7fb0fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.932+0000 7fb0fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 1 -- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb1000a8a30 msgr2=0x7fb1000afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb1000a8a30 0x7fb1000afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 1 -- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb100142a20 msgr2=0x7fb10013b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb100142a20 0x7fb10013b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 1 monclient: found mon.noname-b 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.934+0000 7fb0f7fff640 1 -- 172.21.15.97:0/688747607 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f0009400 con 0x7fb1000a8e10 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.936+0000 7fb0f7fff640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8e10 0x7fb1000aedd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800c400 tx=0x7fb0e800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.936+0000 7fb0f6ffd640 10 client.?.objecter ms_handle_connect 0x7fb1000a8e10 2023-12-07T07:33:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.936+0000 7fb0f6ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 1 -- 172.21.15.97:0/688747607 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8005ce0 con 0x7fb1000a8e10 2023-12-07T07:33:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 10 monclient: dump: 2023-12-07T07:33:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 1 -- 172.21.15.97:0/688747607 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e8005e40 con 0x7fb1000a8e10 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb104f41640 5 monclient: authenticate success, global_id 4402 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb104f41640 10 monclient: _renew_subs 2023-12-07T07:33:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb104f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1000b0020 con 0x7fb1000a8e10 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0f6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb0fd59a640 4 set_mon_vals no callback set 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.939+0000 7fb104f41640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 10 monclient: _renew_subs 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb0f6ffd640 1 -- 172.21.15.97:0/688747607 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e80072d0 con 0x7fb1000a8e10 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10013bde0 con 0x7fb1000a8e10 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 1 librados: init done 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb0f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb0f6ffd640 10 monclient: dump: 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:49.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.940+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb100067ac0 con 0x7fb1000a8e10 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.949+0000 7fd4b08da640 10 monclient: get_monmap_and_config 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.949+0000 7fd4b08da640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.949+0000 7fd4b08da640 10 monclient: monmap: 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:49.950214+0000 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:49.950214+0000 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:49.952 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:49.952 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:49.952 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:49.952 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.949+0000 7fd4b08da640 1 Processor -- start 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.949+0000 7fd4b08da640 1 -- start start 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.949+0000 7fd4b08da640 10 monclient: init 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 0x7fd4ac13ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ac0a8910 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4ac0a9b70 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:49.953 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd4ac141f90 0x7fd4ac1423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ac141f90 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0a8cf0 con 0x7fd4ac0a9b70 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0aabd0 con 0x7fd4ac141f90 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0aaeb0 con 0x7fd4ac0a8910 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.950+0000 7fd4b08da640 10 monclient(hunting): authenticate will time out at 1352.328864s 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.951+0000 7fd4aad9d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd4ac141f90 0x7fd4ac1423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.951+0000 7fd4a9d9b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd4ac0a9b70 0x7fd4ac13f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41384/0 (socket says 172.21.15.97:41384) 2023-12-07T07:33:49.954 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.951+0000 7fd4a9d9b640 1 -- 172.21.15.97:0/3322173087 learned_addr learned my addr 172.21.15.97:0/3322173087 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:49.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.951+0000 7fd4aad9d640 10 monclient(hunting): get_auth_request con 0x7fd4ac141f90 auth_method 0 2023-12-07T07:33:49.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.951+0000 7fd4aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.951+0000 7fd4aad9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.951+0000 7fd4aad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.952+0000 7fd4aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.952+0000 7fd4aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.955 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.952+0000 7fd4aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.957 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 1 -- 172.21.15.97:0/3322173087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 msgr2=0x7fd4ac13ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 1 --2- 172.21.15.97:0/3322173087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 0x7fd4ac13ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 1 -- 172.21.15.97:0/3322173087 >> v1:172.21.15.97:6789/0 conn(0x7fd4ac0a9b70 legacy=0x7fd4ac13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 1 monclient: found mon.noname-a 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.955+0000 7fd4aad9d640 1 -- 172.21.15.97:0/3322173087 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0a9720 con 0x7fd4ac141f90 2023-12-07T07:33:49.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.958+0000 7fb0f6ffd640 1 -- 172.21.15.97:0/688747607 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb0e80074f0 con 0x7fb1000a8e10 2023-12-07T07:33:49.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.958+0000 7fb0f6ffd640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb0e006c7d0 0x7fb0e006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.958+0000 7fb0fe59c640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb0e006c7d0 0x7fb0e006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.958+0000 7fb0fe59c640 10 monclient: get_auth_request con 0x7fb0e006c7d0 auth_method 0 2023-12-07T07:33:49.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.959+0000 7fb0fe59c640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb0e006c7d0 0x7fb0e006ec90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0036000 tx=0x7fb0f003b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.959+0000 7fb0f6ffd640 10 client.4402.objecter ms_handle_connect 0x7fb0e006c7d0 2023-12-07T07:33:49.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.959+0000 7fd4aad9d640 1 --2- 172.21.15.97:0/3322173087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac1423b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009d90 tx=0x7fd4a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f9ff2e09d8f479 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.960+0000 7fb0f6ffd640 1 -- 172.21.15.97:0/688747607 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb0e8057c70 con 0x7fb1000a8e10 2023-12-07T07:33:49.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.960+0000 7fb0f6ffd640 10 client.4402.objecter ms_dispatch 0x7fb100000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:49.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.960+0000 7fb0f6ffd640 3 client.4402.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:49.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.960+0000 7fb0f6ffd640 3 client.4402.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:49.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.960+0000 7fb0f6ffd640 20 client.4402.objecter dump_active .. 0 homeless 2023-12-07T07:33:49.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 1 -- 172.21.15.97:0/3322173087 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a402cc50 con 0x7fd4ac141f90 2023-12-07T07:33:49.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:49.965 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: dump: 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 1 -- 172.21.15.97:0/3322173087 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a402cdb0 con 0x7fd4ac141f90 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4b08da640 5 monclient: authenticate success, global_id 4359 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4b08da640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4b08da640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4b08da640 4 set_mon_vals no callback set 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 1 -- 172.21.15.97:0/3322173087 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4034660 con 0x7fd4ac141f90 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.963+0000 7fd4a959a640 10 monclient: dump: 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.968 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 10 monclient: shutdown 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 -- 172.21.15.97:0/3322173087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 msgr2=0x7fd4ac1423b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3322173087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac1423b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009d90 tx=0x7fd4a402bea0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 -- 172.21.15.97:0/3322173087 shutdown_connections 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3322173087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac1423b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3322173087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 0x7fd4ac13ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 -- 172.21.15.97:0/3322173087 >> 172.21.15.97:0/3322173087 conn(0x7fd4ac0b7780 msgr2=0x7fd4ac0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 -- 172.21.15.97:0/3322173087 shutdown_connections 2023-12-07T07:33:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7fd4b08da640 1 -- 172.21.15.97:0/3322173087 wait complete. 2023-12-07T07:33:49.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7f07b17fa640 1 -- 172.21.15.97:0/576319231 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f079c003ca0 con 0x7f07b40a8a30 2023-12-07T07:33:49.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.964+0000 7f07b17fa640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f079406c820 0x7f079406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7f07b3fff640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f079406c820 0x7f079406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7f07b3fff640 10 monclient: get_auth_request con 0x7f079406c820 auth_method 0 2023-12-07T07:33:49.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7fd4b08da640 10 monclient: build_initial_monmap 2023-12-07T07:33:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7f07b3fff640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f079406c820 0x7f079406ece0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07a4006a90 tx=0x7f07a4006a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7f07b17fa640 10 client.4343.objecter ms_handle_connect 0x7f079406c820 2023-12-07T07:33:49.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7fd4b08da640 10 monclient: monmap: 2023-12-07T07:33:49.972 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7fd4b08da640 1 librados: starting msgr at 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.965+0000 7fd4b08da640 1 librados: starting objecter 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 Processor -- start 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 -- start start 2023-12-07T07:33:49.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 librados: setting wanted keys 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 librados: calling monclient init 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient: init 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 0x7fd4ac141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): picked mon.noname-c con 0x7fd4ac0a8910 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac13a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ac141f90 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4ac13b2a0 0x7fd4ac13b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ac13b2a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac13ae70 con 0x7fd4ac141f90 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0abbc0 con 0x7fd4ac13b2a0 2023-12-07T07:33:49.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0abe30 con 0x7fd4ac0a8910 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.966+0000 7fd4b08da640 10 monclient(hunting): authenticate will time out at 1352.344973s 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac13a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac13a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39232/0 (socket says 172.21.15.97:39232) 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 1 -- 172.21.15.97:0/3700198697 learned_addr learned my addr 172.21.15.97:0/3700198697 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 10 monclient(hunting): get_auth_request con 0x7fd4ac141f90 auth_method 0 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:49.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.967+0000 7fd4a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 1 -- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 msgr2=0x7fd4ac141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 0x7fd4ac141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 1 -- 172.21.15.97:0/3700198697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4ac13b2a0 msgr2=0x7fd4ac13b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4ac13b2a0 0x7fd4ac13b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 10 monclient: _finish_hunting 0 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 1 monclient: found mon.noname-a 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.971+0000 7fd4a9d9b640 1 -- 172.21.15.97:0/3700198697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a4009a70 con 0x7fd4ac141f90 2023-12-07T07:33:49.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.973+0000 7fd4a9d9b640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac13a750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c430 tx=0x7fd49c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.973+0000 7fd49affd640 10 client.?.objecter ms_handle_connect 0x7fd4ac141f90 2023-12-07T07:33:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.973+0000 7fd49affd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:49.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd49affd640 1 -- 172.21.15.97:0/3700198697 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c005ce0 con 0x7fd4ac141f90 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd49affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd49affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd49affd640 10 monclient: dump: 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd49affd640 10 monclient: _finish_auth 0 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd49affd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd49affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 5 monclient: authenticate success, global_id 4365 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 10 monclient: _renew_subs 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac0ac110 con 0x7fd4ac141f90 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 10 monclient: _renew_subs 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac1da8b0 con 0x7fd4ac141f90 2023-12-07T07:33:49.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.975+0000 7fd4b08da640 1 librados: init done 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.976+0000 7fd4b08da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.976+0000 7fd4b08da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.976+0000 7fd4b08da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.976+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac075b60 con 0x7fd4ac141f90 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.977+0000 7fd49affd640 1 -- 172.21.15.97:0/3700198697 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c005e40 con 0x7fd4ac141f90 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.977+0000 7fd49affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.977+0000 7fd49affd640 1 -- 172.21.15.97:0/3700198697 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c004a00 con 0x7fd4ac141f90 2023-12-07T07:33:49.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.977+0000 7fd49affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.977+0000 7fd49affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.977+0000 7fd49affd640 10 monclient: dump: 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:49.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.977+0000 7fd4a8d99640 4 set_mon_vals no callback set 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 1 -- 172.21.15.97:0/576319231 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f079c0010c0 con 0x7f07b40a8a30 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 10 client.4343.objecter ms_dispatch 0x7f07b4000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 3 client.4343.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 3 client.4343.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 20 client.4343.objecter dump_active .. 0 homeless 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 1 -- 172.21.15.97:0/576319231 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f079c054290 con 0x7f07b40a8a30 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:49.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:49.989+0000 7f07b17fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.116+0000 7fc55f184640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:50.119 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.116+0000 7fc55f184640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:50.119 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.116+0000 7fc55f184640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:50.119 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.116+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc558075b60 con 0x7fc5580a8e10 2023-12-07T07:33:50.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.167+0000 7f07bac75640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:50.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.167+0000 7f07bac75640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:50.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.167+0000 7f07bac75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:50.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.167+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f07b4144250 con 0x7f07b40a8a30 2023-12-07T07:33:50.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.174+0000 7f07b17fa640 1 -- 172.21.15.97:0/576319231 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f079c017090 con 0x7f07b40a8a30 2023-12-07T07:33:50.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.174+0000 7f07b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-07T07:33:50.177 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.174+0000 7f07b17fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:50.177 INFO:teuthology.orchestra.run.smithi097.stdout:55834574855 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 10 librados: watch_flush enter 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 10 librados: watch_flush exit 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 20 client.4343.objecter shutdown clearing up homeless session... 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 10 client.4343.objecter successfully canceled tick 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f079406c820 msgr2=0x7f079406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f079406c820 0x7f079406ece0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07a4006a90 tx=0x7f07a4006a20 comp rx=0 tx=0).stop 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 10 monclient: shutdown 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:50.179 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8a30 msgr2=0x7f07b40abb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8a30 0x7f07b40abb70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f079c00c410 tx=0x7f079c00c8d0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 shutdown_connections 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f079406c820 0x7f079406ece0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07b4142bf0 0x7f07b41daa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.176+0000 7f07bac75640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b40a8e10 0x7f07b40ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.177+0000 7f07bac75640 1 --2- 172.21.15.97:0/576319231 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07b40a8a30 0x7f07b40abb70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.177+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 >> 172.21.15.97:0/576319231 conn(0x7f07b4061a30 msgr2=0x7f07b40b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.177+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 shutdown_connections 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.177+0000 7f07bac75640 1 -- 172.21.15.97:0/576319231 wait complete. 2023-12-07T07:33:50.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.177+0000 7f07bac75640 1 librados: shutdown 2023-12-07T07:33:50.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.7 2023-12-07T07:33:50.190 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:50.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.199+0000 7fc5557fa640 1 -- 172.21.15.97:0/1949861510 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc544079eb0 con 0x7fc5580a8e10 2023-12-07T07:33:50.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.199+0000 7fc5557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-07T07:33:50.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.199+0000 7fc5557fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:50.202 INFO:teuthology.orchestra.run.smithi097.stdout:73014444038 2023-12-07T07:33:50.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.200+0000 7fb0f6ffd640 1 -- 172.21.15.97:0/688747607 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0e8053fb0 con 0x7fb1000a8e10 2023-12-07T07:33:50.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.200+0000 7fb0f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:50.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.200+0000 7fb0f6ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:50.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 10 librados: watch_flush enter 2023-12-07T07:33:50.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 10 librados: watch_flush exit 2023-12-07T07:33:50.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 20 client.4393.objecter shutdown clearing up homeless session... 2023-12-07T07:33:50.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 10 client.4393.objecter successfully canceled tick 2023-12-07T07:33:50.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc53c06c820 msgr2=0x7fc53c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc53c06c820 0x7fc53c06ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0017f0 tx=0x7fc54c009040 comp rx=0 tx=0).stop 2023-12-07T07:33:50.204 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 10 monclient: shutdown 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5580a8e10 msgr2=0x7fc5580aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5580a8e10 0x7fc5580aea20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5440090b0 tx=0x7fc5440046c0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 shutdown_connections 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc53c06c820 0x7fc53c06ece0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc558142bf0 0x7fc5581da9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5580a8e10 0x7fc5580aea20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 --2- 172.21.15.97:0/1949861510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc5580a8a30 0x7fc5580adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 >> 172.21.15.97:0/1949861510 conn(0x7fc558061a30 msgr2=0x7fc55813cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 shutdown_connections 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 -- 172.21.15.97:0/1949861510 wait complete. 2023-12-07T07:33:50.205 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.201+0000 7fc55f184640 1 librados: shutdown 2023-12-07T07:33:50.215 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444038 for osd.1 2023-12-07T07:33:50.215 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:50.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.235+0000 7fd49affd640 1 -- 172.21.15.97:0/3700198697 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd49c007070 con 0x7fd4ac141f90 2023-12-07T07:33:50.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.235+0000 7fd49affd640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd48c06c7d0 0x7fd48c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:50.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.236+0000 7fd4aa59c640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd48c06c7d0 0x7fd48c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:50.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.236+0000 7fd4aa59c640 10 monclient: get_auth_request con 0x7fd48c06c7d0 auth_method 0 2023-12-07T07:33:50.238 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.236+0000 7fd49affd640 10 client.4365.objecter ms_handle_connect 0x7fd48c06c7d0 2023-12-07T07:33:50.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.236+0000 7fd4aa59c640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd48c06c7d0 0x7fd48c06ec90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009710 tx=0x7fd4a0006c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:50.253 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 1 -- 172.21.15.97:0/3700198697 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd49c057d70 con 0x7fd4ac141f90 2023-12-07T07:33:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 10 client.4365.objecter ms_dispatch 0x7fd4ac000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 3 client.4365.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 3 client.4365.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 20 client.4365.objecter dump_active .. 0 homeless 2023-12-07T07:33:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 1 -- 172.21.15.97:0/3700198697 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd49c0540b0 con 0x7fd4ac141f90 2023-12-07T07:33:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:50.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.251+0000 7fd49affd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:50.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.415+0000 7fb104f41640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:50.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.415+0000 7fb104f41640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:50.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.415+0000 7fb104f41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:50.417 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.415+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb100142eb0 con 0x7fb1000a8e10 2023-12-07T07:33:50.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.418+0000 7fb0f6ffd640 1 -- 172.21.15.97:0/688747607 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb100142eb0 con 0x7fb1000a8e10 2023-12-07T07:33:50.420 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.418+0000 7fb0f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-07T07:33:50.421 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.418+0000 7fb0f6ffd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:50.421 INFO:teuthology.orchestra.run.smithi097.stdout:55834574853 2023-12-07T07:33:50.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 10 librados: watch_flush enter 2023-12-07T07:33:50.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 10 librados: watch_flush exit 2023-12-07T07:33:50.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 20 client.4402.objecter shutdown clearing up homeless session... 2023-12-07T07:33:50.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 10 client.4402.objecter successfully canceled tick 2023-12-07T07:33:50.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb0e006c7d0 msgr2=0x7fb0e006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb0e006c7d0 0x7fb0e006ec90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0036000 tx=0x7fb0f003b040 comp rx=0 tx=0).stop 2023-12-07T07:33:50.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 10 monclient: shutdown 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8e10 msgr2=0x7fb1000aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8e10 0x7fb1000aedd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800c400 tx=0x7fb0e800c8c0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 shutdown_connections 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb0e006c7d0 0x7fb0e006ec90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb100142a20 0x7fb10013b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1000a8e10 0x7fb1000aedd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.420+0000 7fb104f41640 1 --2- 172.21.15.97:0/688747607 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb1000a8a30 0x7fb1000afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.421+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 >> 172.21.15.97:0/688747607 conn(0x7fb100061a30 msgr2=0x7fb1000b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.421+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 shutdown_connections 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.421+0000 7fb104f41640 1 -- 172.21.15.97:0/688747607 wait complete. 2023-12-07T07:33:50.424 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.421+0000 7fb104f41640 1 librados: shutdown 2023-12-07T07:33:50.436 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2023-12-07T07:33:50.436 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:50.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.449+0000 7fd4b08da640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:50.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.449+0000 7fd4b08da640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:50.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.449+0000 7fd4b08da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:50.451 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.449+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd4ac13e320 con 0x7fd4ac141f90 2023-12-07T07:33:50.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.479+0000 7fd49affd640 1 -- 172.21.15.97:0/3700198697 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd49c017020 con 0x7fd4ac141f90 2023-12-07T07:33:50.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.479+0000 7fd49affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-07T07:33:50.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.479+0000 7fd49affd640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:50.482 INFO:teuthology.orchestra.run.smithi097.stdout:73014444038 2023-12-07T07:33:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 10 librados: watch_flush enter 2023-12-07T07:33:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 10 librados: watch_flush exit 2023-12-07T07:33:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 20 client.4365.objecter shutdown clearing up homeless session... 2023-12-07T07:33:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 10 client.4365.objecter successfully canceled tick 2023-12-07T07:33:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd48c06c7d0 msgr2=0x7fd48c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd48c06c7d0 0x7fd48c06ec90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009710 tx=0x7fd4a0006c90 comp rx=0 tx=0).stop 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 10 monclient: shutdown 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 msgr2=0x7fd4ac13a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac13a750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c430 tx=0x7fd49c00c8f0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 shutdown_connections 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd48c06c7d0 0x7fd48c06ec90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4ac13b2a0 0x7fd4ac13b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd4ac141f90 0x7fd4ac13a750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 --2- 172.21.15.97:0/3700198697 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd4ac0a8910 0x7fd4ac141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 >> 172.21.15.97:0/3700198697 conn(0x7fd4ac0b7780 msgr2=0x7fd4ac0b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 shutdown_connections 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 -- 172.21.15.97:0/3700198697 wait complete. 2023-12-07T07:33:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.481+0000 7fd4b08da640 1 librados: shutdown 2023-12-07T07:33:50.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444038 for osd.3 2023-12-07T07:33:50.495 DEBUG:teuthology.parallel:result is None 2023-12-07T07:33:50.495 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-07T07:33:50.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:33:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.632+0000 7f2f1e71d640 10 monclient: get_monmap_and_config 2023-12-07T07:33:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.632+0000 7f2f1e71d640 10 monclient: build_initial_monmap 2023-12-07T07:33:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.632+0000 7f2f1e71d640 10 monclient: monmap: 2023-12-07T07:33:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:50.634 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:50.633221+0000 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:50.633221+0000 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:50.635 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:50.636 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.632+0000 7f2f1e71d640 1 Processor -- start 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.632+0000 7f2f1e71d640 1 -- start start 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.632+0000 7f2f1e71d640 10 monclient: init 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f180a8a30 addr v1:172.21.15.97:6789/0 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f18142bf0 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2f180a8e10 0x7f2f1813f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f180a8e10 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f180aac10 con 0x7f2f180a8a30 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f180aaef0 con 0x7f2f18142bf0 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f180aaa10 con 0x7f2f180a8e10 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:50.637 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.633+0000 7f2f1e71d640 10 monclient(hunting): authenticate will time out at 1353.011827s 2023-12-07T07:33:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.634+0000 7f2f1ccba640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2f180a8e10 0x7f2f1813f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.634+0000 7f2f177fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f2f18142bf0 0x7f2f1813ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:59118/0 (socket says 172.21.15.97:59118) 2023-12-07T07:33:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.634+0000 7f2f177fe640 1 -- 172.21.15.97:0/752492154 learned_addr learned my addr 172.21.15.97:0/752492154 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:33:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.634+0000 7f2f1ccba640 10 monclient(hunting): get_auth_request con 0x7f2f180a8e10 auth_method 0 2023-12-07T07:33:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.634+0000 7f2f1ccba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.634+0000 7f2f1ccba640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.634+0000 7f2f1ccba640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:50.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.635+0000 7f2f1ccba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:50.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.635+0000 7f2f1ccba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.635+0000 7f2f1ccba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.636+0000 7f2f16ffd640 1 -- 172.21.15.97:0/752492154 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258870340 0 0) 0x7f2f180aaef0 con 0x7f2f18142bf0 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.636+0000 7f2f16ffd640 10 monclient(hunting): _negotiate 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.636+0000 7f2f16ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.636+0000 7f2f16ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.636+0000 7f2f16ffd640 10 monclient(hunting): my global_id is 4405 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.636+0000 7f2f16ffd640 1 -- 172.21.15.97:0/752492154 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f00003520 con 0x7f2f18142bf0 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 1 -- 172.21.15.97:0/752492154 >> v1:172.21.15.154:6789/0 conn(0x7f2f18142bf0 legacy=0x7f2f1813ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 1 -- 172.21.15.97:0/752492154 >> v1:172.21.15.97:6789/0 conn(0x7f2f180a8a30 legacy=0x7f2f181424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 10 monclient: _finish_hunting 0 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 1 monclient: found mon.noname-a 2023-12-07T07:33:50.640 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:50.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:50.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.637+0000 7f2f1ccba640 1 -- 172.21.15.97:0/752492154 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f180a97f0 con 0x7f2f180a8e10 2023-12-07T07:33:50.643 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.641+0000 7f2f1ccba640 1 --2- 172.21.15.97:0/752492154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f1813f400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009d90 tx=0x7f2f1002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1937387a8c2ff2 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:50.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 1 -- 172.21.15.97:0/752492154 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1002cd80 con 0x7f2f180a8e10 2023-12-07T07:33:50.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:50.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:50.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: dump: 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:50.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 1 -- 172.21.15.97:0/752492154 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f10034440 con 0x7f2f180a8e10 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f1e71d640 5 monclient: authenticate success, global_id 4368 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f1e71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f1e71d640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f1e71d640 4 set_mon_vals no callback set 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 1 -- 172.21.15.97:0/752492154 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f100347d0 con 0x7f2f180a8e10 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.645+0000 7f2f16ffd640 10 monclient: dump: 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:50.649 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:50.650 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:50.651 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:50.651 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:50.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 10 monclient: shutdown 2023-12-07T07:33:50.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:50.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 1 -- 172.21.15.97:0/752492154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 msgr2=0x7f2f1813f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/752492154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f1813f400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009d90 tx=0x7f2f1002bfd0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.655 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 1 -- 172.21.15.97:0/752492154 shutdown_connections 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/752492154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f1813f400 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 1 -- 172.21.15.97:0/752492154 >> 172.21.15.97:0/752492154 conn(0x7f2f18061a30 msgr2=0x7f2f180b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 1 -- 172.21.15.97:0/752492154 shutdown_connections 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 1 -- 172.21.15.97:0/752492154 wait complete. 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 10 monclient: build_initial_monmap 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.646+0000 7f2f1e71d640 10 monclient: monmap: 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.647+0000 7f2f1e71d640 1 librados: starting msgr at 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.647+0000 7f2f1e71d640 1 librados: starting objecter 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.647+0000 7f2f1e71d640 1 Processor -- start 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.647+0000 7f2f1e71d640 1 -- start start 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.647+0000 7f2f1e71d640 1 librados: setting wanted keys 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.647+0000 7f2f1e71d640 1 librados: calling monclient init 2023-12-07T07:33:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.647+0000 7f2f1e71d640 10 monclient: init 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2f180a8a30 0x7f2f18144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): picked mon.noname-c con 0x7f2f180a8a30 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f180b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f180a8e10 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f18142bf0 0x7f2f180b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): picked mon.noname-b con 0x7f2f18142bf0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180b64e0 con 0x7f2f180a8e10 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f181461e0 con 0x7f2f18142bf0 2023-12-07T07:33:50.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18143240 con 0x7f2f180a8a30 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1e71d640 10 monclient(hunting): authenticate will time out at 1353.026392s 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f177fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f180b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2f180a8a30 0x7f2f18144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f177fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f180b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47944/0 (socket says 172.21.15.97:47944) 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f177fe640 1 -- 172.21.15.97:0/393348800 learned_addr learned my addr 172.21.15.97:0/393348800 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f177fe640 10 monclient(hunting): get_auth_request con 0x7f2f180a8e10 auth_method 0 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:50.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f17fff640 10 monclient(hunting): get_auth_request con 0x7f2f180a8a30 auth_method 0 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f17fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f17fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.648+0000 7f2f1ccba640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f18142bf0 0x7f2f180b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f1ccba640 10 monclient(hunting): get_auth_request con 0x7f2f18142bf0 auth_method 0 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f1ccba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f1ccba640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f1ccba640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:50.660 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f1ccba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f1ccba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.649+0000 7f2f1ccba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.651+0000 7f2f177fe640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 1 -- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2f180a8a30 msgr2=0x7f2f18144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2f180a8a30 0x7f2f18144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 1 -- 172.21.15.97:0/393348800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f18142bf0 msgr2=0x7f2f180b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f18142bf0 0x7f2f180b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 1 monclient: found mon.noname-a 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:50.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.652+0000 7f2f177fe640 1 -- 172.21.15.97:0/393348800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f10009a70 con 0x7f2f180a8e10 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.654+0000 7f2f177fe640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f180b5a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00c3e0 tx=0x7f2f0c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.654+0000 7f2f14ff9640 10 client.?.objecter ms_handle_connect 0x7f2f180a8e10 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.654+0000 7f2f14ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 1 -- 172.21.15.97:0/393348800 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c005ce0 con 0x7f2f180a8e10 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: dump: 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: _finish_auth 0 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 1 -- 172.21.15.97:0/393348800 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c005e40 con 0x7f2f180a8e10 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 1 -- 172.21.15.97:0/393348800 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0072d0 con 0x7f2f180a8e10 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 5 monclient: authenticate success, global_id 4374 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 10 monclient: _renew_subs 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:50.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f18144dd0 con 0x7f2f180a8e10 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f14ff9640 10 monclient: dump: 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f167fc640 4 set_mon_vals no callback set 2023-12-07T07:33:50.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 10 monclient: _renew_subs 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f181445a0 con 0x7f2f180a8e10 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.657+0000 7f2f1e71d640 1 librados: init done 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.658+0000 7f2f1e71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.658+0000 7f2f1e71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.658+0000 7f2f1e71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:33:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.658+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f18067ac0 con 0x7f2f180a8e10 2023-12-07T07:33:50.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.676+0000 7f2f14ff9640 1 -- 172.21.15.97:0/393348800 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2f0c0074f0 con 0x7f2f180a8e10 2023-12-07T07:33:50.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.676+0000 7f2f14ff9640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2f0406c7d0 0x7f2f0406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:50.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.676+0000 7f2f17fff640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2f0406c7d0 0x7f2f0406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:50.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.677+0000 7f2f17fff640 10 monclient: get_auth_request con 0x7f2f0406c7d0 auth_method 0 2023-12-07T07:33:50.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.677+0000 7f2f14ff9640 10 client.4374.objecter ms_handle_connect 0x7f2f0406c7d0 2023-12-07T07:33:50.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.677+0000 7f2f17fff640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2f0406c7d0 0x7f2f0406ec90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f08001780 tx=0x7f2f08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:50.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.679+0000 7f2f14ff9640 1 -- 172.21.15.97:0/393348800 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2f0c057e50 con 0x7f2f180a8e10 2023-12-07T07:33:50.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.679+0000 7f2f14ff9640 10 client.4374.objecter ms_dispatch 0x7f2f18000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:50.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.679+0000 7f2f14ff9640 3 client.4374.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:50.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.679+0000 7f2f14ff9640 3 client.4374.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:50.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.679+0000 7f2f14ff9640 20 client.4374.objecter dump_active .. 0 homeless 2023-12-07T07:33:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.911+0000 7f2f14ff9640 1 -- 172.21.15.97:0/393348800 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f0c054190 con 0x7f2f180a8e10 2023-12-07T07:33:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.911+0000 7f2f14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:50.911+0000 7f2f14ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:51.079 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.076+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f1813f940 con 0x7f2f0406c7d0 2023-12-07T07:33:51.081 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.078+0000 7f2f14ff9640 1 -- 172.21.15.97:0/393348800 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48200 (secure 0 0 0) 0x7f2f08001420 con 0x7f2f0406c7d0 2023-12-07T07:33:51.081 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:33:51.081 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:33:51.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 10 librados: watch_flush enter 2023-12-07T07:33:51.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 10 librados: watch_flush exit 2023-12-07T07:33:51.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 20 client.4374.objecter shutdown clearing up homeless session... 2023-12-07T07:33:51.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 10 client.4374.objecter successfully canceled tick 2023-12-07T07:33:51.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2f0406c7d0 msgr2=0x7f2f0406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2f0406c7d0 0x7f2f0406ec90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f08001780 tx=0x7f2f08009040 comp rx=0 tx=0).stop 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 10 monclient: shutdown 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 msgr2=0x7f2f180b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.080+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f180b5a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00c3e0 tx=0x7f2f0c00c8a0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 shutdown_connections 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2f0406c7d0 0x7f2f0406ec90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f18142bf0 0x7f2f180b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.085 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f180a8e10 0x7f2f180b5a60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 --2- 172.21.15.97:0/393348800 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2f180a8a30 0x7f2f18144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 >> 172.21.15.97:0/393348800 conn(0x7f2f18061a30 msgr2=0x7f2f1813af50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:51.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 shutdown_connections 2023-12-07T07:33:51.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 -- 172.21.15.97:0/393348800 wait complete. 2023-12-07T07:33:51.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.081+0000 7f2f1e71d640 1 librados: shutdown 2023-12-07T07:33:51.094 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-12-07T07:33:49.625123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218976,"kb_used_data":2624,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755744,"statfs":{"total":773094113280,"available":772869881856,"internally_reserved":0,"allocated":2686976,"data_stored":1476087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.673856"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.771097+0000","last_change":"2023-12-07T07:33:42.670811+0000","last_active":"2023-12-07T07:33:44.771097+0000","last_peered":"2023-12-07T07:33:44.771097+0000","last_clean":"2023-12-07T07:33:44.771097+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:33:44.771097+0000","last_undegraded":"2023-12-07T07:33:44.771097+0000","last_fullsized":"2023-12-07T07:33:44.771097+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:33:38.367199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:45.066712+0000","last_change":"2023-12-07T07:33:43.216945+0000","last_active":"2023-12-07T07:33:45.066712+0000","last_peered":"2023-12-07T07:33:45.066712+0000","last_clean":"2023-12-07T07:33:45.066712+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:33:45.066712+0000","last_undegraded":"2023-12-07T07:33:45.066712+0000","last_fullsized":"2023-12-07T07:33:45.066712+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:54.221713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.768489+0000","last_change":"2023-12-07T07:33:42.426643+0000","last_active":"2023-12-07T07:33:44.768489+0000","last_peered":"2023-12-07T07:33:44.768489+0000","last_clean":"2023-12-07T07:33:44.768489+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:33:44.768489+0000","last_undegraded":"2023-12-07T07:33:44.768489+0000","last_fullsized":"2023-12-07T07:33:44.768489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:11:45.155210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:45.266183+0000","last_change":"2023-12-07T07:33:42.656077+0000","last_active":"2023-12-07T07:33:45.266183+0000","last_peered":"2023-12-07T07:33:45.266183+0000","last_clean":"2023-12-07T07:33:45.266183+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:33:45.266183+0000","last_undegraded":"2023-12-07T07:33:45.266183+0000","last_fullsized":"2023-12-07T07:33:45.266183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:05:05.686340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.759576+0000","last_change":"2023-12-07T07:33:42.617026+0000","last_active":"2023-12-07T07:33:44.759576+0000","last_peered":"2023-12-07T07:33:44.759576+0000","last_clean":"2023-12-07T07:33:44.759576+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:33:44.759576+0000","last_undegraded":"2023-12-07T07:33:44.759576+0000","last_fullsized":"2023-12-07T07:33:44.759576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:58.336948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:45.244239+0000","last_change":"2023-12-07T07:33:42.269533+0000","last_active":"2023-12-07T07:33:45.244239+0000","last_peered":"2023-12-07T07:33:45.244239+0000","last_clean":"2023-12-07T07:33:45.244239+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:33:45.244239+0000","last_undegraded":"2023-12-07T07:33:45.244239+0000","last_fullsized":"2023-12-07T07:33:45.244239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:12:50.238470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.750832+0000","last_change":"2023-12-07T07:33:42.349265+0000","last_active":"2023-12-07T07:33:44.750832+0000","last_peered":"2023-12-07T07:33:44.750832+0000","last_clean":"2023-12-07T07:33:44.750832+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:33:44.750832+0000","last_undegraded":"2023-12-07T07:33:44.750832+0000","last_fullsized":"2023-12-07T07:33:44.750832+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:03:37.867596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.762583+0000","last_change":"2023-12-07T07:33:42.649680+0000","last_active":"2023-12-07T07:33:44.762583+0000","last_peered":"2023-12-07T07:33:44.762583+0000","last_clean":"2023-12-07T07:33:44.762583+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:33:44.762583+0000","last_undegraded":"2023-12-07T07:33:44.762583+0000","last_fullsized":"2023-12-07T07:33:44.762583+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:13:55.895630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:46.196187+0000","last_change":"2023-12-07T07:33:36.384672+0000","last_active":"2023-12-07T07:33:46.196187+0000","last_peered":"2023-12-07T07:33:46.196187+0000","last_clean":"2023-12-07T07:33:46.196187+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:33:46.196187+0000","last_undegraded":"2023-12-07T07:33:46.196187+0000","last_fullsized":"2023-12-07T07:33:46.196187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_clean_scrub_stamp":"2023-12-07T07:33:30.679434+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:54:14.180629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444038,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1150000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.056000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.751999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.946000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3079999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.385000000000002}]}]},{"osd":2,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.766}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8550000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.451000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.922999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.901}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.036000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.902999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":634880,"data_stored":483283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.861999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.968}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.183}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.574}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":46.597999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.531000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":52.234999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.808}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.09}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1649999999999991}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.646000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.428999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.647}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.981}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.411}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":45.107999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":271.90699999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.309000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.968}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":80068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":82.552999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":98.188999999999993}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":154.01499999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":482.79399999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.516}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.872}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":483656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.343}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7409999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.839}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.646999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.204999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.337}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.108000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.654}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.852}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.173}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.24}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9779999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.608000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:33:51.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.193+0000 7f2d38de9640 10 monclient: get_monmap_and_config 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.193+0000 7f2d38de9640 10 monclient: build_initial_monmap 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.193+0000 7f2d38de9640 10 monclient: monmap: 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:51.194912+0000 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:51.194912+0000 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:51.196 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:51.197 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:51.197 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:51.197 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.194+0000 7f2d38de9640 1 Processor -- start 2023-12-07T07:33:51.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.194+0000 7f2d38de9640 1 -- start start 2023-12-07T07:33:51.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.194+0000 7f2d38de9640 10 monclient: init 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d340a8950 addr v1:172.21.15.154:6789/0 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2d340a8d30 0x7f2d341457b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient(hunting): picked mon.noname-a con 0x7f2d340a8d30 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d3414c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient(hunting): picked mon.noname-b con 0x7f2d34145cf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d34145320 con 0x7f2d340a8950 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d340b2260 con 0x7f2d340a8d30 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34146180 con 0x7f2d34145cf0 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d38de9640 10 monclient(hunting): authenticate will time out at 1353.573586s 2023-12-07T07:33:51.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d31d9b640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2d340a8d30 0x7f2d341457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.195+0000 7f2d32d9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d3414c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2d340a8d30 0x7f2d341457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47950/0 (socket says 172.21.15.97:47950) 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 1 -- 172.21.15.97:0/1864820341 learned_addr learned my addr 172.21.15.97:0/1864820341 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 10 monclient(hunting): get_auth_request con 0x7f2d340a8d30 auth_method 0 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d32d9d640 10 monclient(hunting): get_auth_request con 0x7f2d34145cf0 auth_method 0 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d32d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d32d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.200 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.196+0000 7f2d32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2023-12-07T07:33:51.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 1 -- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 msgr2=0x7f2d3414c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:51.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 1 --2- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d3414c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 1 -- 172.21.15.97:0/1864820341 >> v1:172.21.15.154:6789/0 conn(0x7f2d340a8950 legacy=0x7f2d3413f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:51.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 10 monclient: _finish_hunting 0 2023-12-07T07:33:51.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 1 monclient: found mon.noname-a 2023-12-07T07:33:51.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:51.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:33:51.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d31d9b640 1 -- 172.21.15.97:0/1864820341 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d340aa5e0 con 0x7f2d340a8d30 2023-12-07T07:33:51.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.198+0000 7f2d32d9d640 1 --2- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d3414c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-07T07:33:51.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.200+0000 7f2d31d9b640 1 --2- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8d30 0x7f2d341457b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009870 tx=0x7f2d2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585f28cd3d645526 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.205+0000 7f2d3159a640 1 -- 172.21.15.97:0/1864820341 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28031640 con 0x7f2d340a8d30 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.205+0000 7f2d3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.205+0000 7f2d3159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.205+0000 7f2d3159a640 10 monclient: dump: 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.208 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.205+0000 7f2d3159a640 10 monclient: _finish_auth 0 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.205+0000 7f2d3159a640 10 monclient: _check_auth_tickets 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.205+0000 7f2d3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 5 monclient: authenticate success, global_id 4377 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d3159a640 1 -- 172.21.15.97:0/1864820341 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d280317a0 con 0x7f2d340a8d30 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d3159a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d3159a640 1 -- 172.21.15.97:0/1864820341 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28031b30 con 0x7f2d340a8d30 2023-12-07T07:33:51.209 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 4 set_mon_vals no callback set 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d3159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d3159a640 10 monclient: dump: 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.210 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.213 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 10 monclient: shutdown 2023-12-07T07:33:51.213 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:51.213 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 -- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8d30 msgr2=0x7f2d341457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.213 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 --2- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8d30 0x7f2d341457b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009870 tx=0x7f2d2802b8f0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 -- 172.21.15.97:0/1864820341 shutdown_connections 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 --2- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d3414c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 --2- 172.21.15.97:0/1864820341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8d30 0x7f2d341457b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 -- 172.21.15.97:0/1864820341 >> 172.21.15.97:0/1864820341 conn(0x7f2d34061970 msgr2=0x7f2d340b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 -- 172.21.15.97:0/1864820341 shutdown_connections 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.206+0000 7f2d38de9640 1 -- 172.21.15.97:0/1864820341 wait complete. 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 10 monclient: build_initial_monmap 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 10 monclient: monmap: 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:51.214 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 1 librados: starting msgr at 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 1 librados: starting objecter 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 1 Processor -- start 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 1 -- start start 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 1 librados: setting wanted keys 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 1 librados: calling monclient init 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.208+0000 7f2d38de9640 10 monclient: init 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:33:51.215 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8950 0x7f2d341cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): picked mon.noname-a con 0x7f2d340a8950 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a8d30 0x7f2d34137d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): picked mon.noname-b con 0x7f2d340a8d30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d34138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): picked mon.noname-c con 0x7f2d34145cf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341387c0 con 0x7f2d340a8950 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341cd1f0 con 0x7f2d340a8d30 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341cd4d0 con 0x7f2d34145cf0 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d38de9640 10 monclient(hunting): authenticate will time out at 1353.587444s 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d32d9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d34138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.216 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d31d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a8d30 0x7f2d34137d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d3259c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8950 0x7f2d341cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d3259c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8950 0x7f2d341cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47952/0 (socket says 172.21.15.97:47952) 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d3259c640 1 -- 172.21.15.97:0/2332051469 learned_addr learned my addr 172.21.15.97:0/2332051469 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d3259c640 10 monclient(hunting): get_auth_request con 0x7f2d340a8950 auth_method 0 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d32d9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d34138280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:44640/0 (socket says 172.21.15.97:44640) 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d3259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d3259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d32d9d640 10 monclient(hunting): get_auth_request con 0x7f2d34145cf0 auth_method 0 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d32d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.209+0000 7f2d32d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d31d9b640 10 monclient(hunting): get_auth_request con 0x7f2d340a8d30 auth_method 0 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d31d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d31d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.210+0000 7f2d31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 1 -- 172.21.15.97:0/2332051469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a8d30 msgr2=0x7f2d34137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a8d30 0x7f2d34137d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 1 -- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8950 msgr2=0x7f2d341cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8950 0x7f2d341cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 10 monclient: _finish_hunting 0 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 1 monclient: found mon.noname-c 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:51.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:33:51.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.213+0000 7f2d32d9d640 1 -- 172.21.15.97:0/2332051469 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c009a70 con 0x7f2d34145cf0 2023-12-07T07:33:51.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.215+0000 7f2d32d9d640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d34138280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00efc0 tx=0x7f2d2c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:51.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.215+0000 7f2d22ffd640 10 client.?.objecter ms_handle_connect 0x7f2d34145cf0 2023-12-07T07:33:51.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.215+0000 7f2d22ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:51.221 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 1 -- 172.21.15.97:0/2332051469 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00cd30 con 0x7f2d34145cf0 2023-12-07T07:33:51.221 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.221 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:51.221 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: dump: 2023-12-07T07:33:51.221 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.221 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: _finish_auth 0 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 1 -- 172.21.15.97:0/2332051469 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c00f040 con 0x7f2d34145cf0 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:51.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 1 -- 172.21.15.97:0/2332051469 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c003bd0 con 0x7f2d34145cf0 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 5 monclient: authenticate success, global_id 4352 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 10 monclient: _renew_subs 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d280094d0 con 0x7f2d34145cf0 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d30d99640 4 set_mon_vals no callback set 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d22ffd640 10 monclient: dump: 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.223 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 10 monclient: _renew_subs 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.218+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d341cdc40 con 0x7f2d34145cf0 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.219+0000 7f2d38de9640 1 librados: init done 2023-12-07T07:33:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.220+0000 7f2d38de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:51.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.220+0000 7f2d38de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:51.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.220+0000 7f2d38de9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:33:51.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.220+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d341cde30 con 0x7f2d34145cf0 2023-12-07T07:33:51.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.237+0000 7f2d22ffd640 1 -- 172.21.15.97:0/2332051469 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2d2c014480 con 0x7f2d34145cf0 2023-12-07T07:33:51.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.237+0000 7f2d22ffd640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2d1406c7d0 0x7f2d1406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.237+0000 7f2d3259c640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2d1406c7d0 0x7f2d1406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.237+0000 7f2d3259c640 10 monclient: get_auth_request con 0x7f2d1406c7d0 auth_method 0 2023-12-07T07:33:51.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.237+0000 7f2d22ffd640 10 client.4352.objecter ms_handle_connect 0x7f2d1406c7d0 2023-12-07T07:33:51.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.237+0000 7f2d3259c640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2d1406c7d0 0x7f2d1406ec90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009990 tx=0x7f2d24009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:51.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.240+0000 7f2d22ffd640 1 -- 172.21.15.97:0/2332051469 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2d2c059640 con 0x7f2d34145cf0 2023-12-07T07:33:51.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.240+0000 7f2d22ffd640 10 client.4352.objecter ms_dispatch 0x7f2d34000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:51.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.240+0000 7f2d22ffd640 3 client.4352.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:51.242 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.240+0000 7f2d22ffd640 3 client.4352.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:51.243 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.240+0000 7f2d22ffd640 20 client.4352.objecter dump_active .. 0 homeless 2023-12-07T07:33:51.481 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.479+0000 7f2d22ffd640 1 -- 172.21.15.97:0/2332051469 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d2c055980 con 0x7f2d34145cf0 2023-12-07T07:33:51.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.479+0000 7f2d22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:51.482 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.479+0000 7f2d22ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:51.659 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.657+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d340aae80 con 0x7f2d1406c7d0 2023-12-07T07:33:51.661 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.659+0000 7f2d22ffd640 1 -- 172.21.15.97:0/2332051469 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48193 (secure 0 0 0) 0x7f2d2400e850 con 0x7f2d1406c7d0 2023-12-07T07:33:51.661 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:33:51.662 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 10 librados: watch_flush enter 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 10 librados: watch_flush exit 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 20 client.4352.objecter shutdown clearing up homeless session... 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 10 client.4352.objecter successfully canceled tick 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2d1406c7d0 msgr2=0x7f2d1406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2d1406c7d0 0x7f2d1406ec90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009990 tx=0x7f2d24009110 comp rx=0 tx=0).stop 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 10 monclient: shutdown 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:51.664 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 msgr2=0x7f2d34138280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d34138280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00efc0 tx=0x7f2d2c00c490 comp rx=0 tx=0).stop 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 shutdown_connections 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2d1406c7d0 0x7f2d1406ec90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2d34145cf0 0x7f2d34138280 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a8d30 0x7f2d34137d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 --2- 172.21.15.97:0/2332051469 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d340a8950 0x7f2d341cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 >> 172.21.15.97:0/2332051469 conn(0x7f2d34061970 msgr2=0x7f2d340b5450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 shutdown_connections 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 -- 172.21.15.97:0/2332051469 wait complete. 2023-12-07T07:33:51.665 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.661+0000 7f2d38de9640 1 librados: shutdown 2023-12-07T07:33:51.674 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-12-07T07:33:51.625437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219172,"kb_used_data":2636,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755548,"statfs":{"total":773094113280,"available":772869681152,"internally_reserved":0,"allocated":2699264,"data_stored":1481835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.614317"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.771097+0000","last_change":"2023-12-07T07:33:42.670811+0000","last_active":"2023-12-07T07:33:44.771097+0000","last_peered":"2023-12-07T07:33:44.771097+0000","last_clean":"2023-12-07T07:33:44.771097+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:33:44.771097+0000","last_undegraded":"2023-12-07T07:33:44.771097+0000","last_fullsized":"2023-12-07T07:33:44.771097+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:33:38.367199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:45.066712+0000","last_change":"2023-12-07T07:33:43.216945+0000","last_active":"2023-12-07T07:33:45.066712+0000","last_peered":"2023-12-07T07:33:45.066712+0000","last_clean":"2023-12-07T07:33:45.066712+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:33:45.066712+0000","last_undegraded":"2023-12-07T07:33:45.066712+0000","last_fullsized":"2023-12-07T07:33:45.066712+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:54.221713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.768489+0000","last_change":"2023-12-07T07:33:42.426643+0000","last_active":"2023-12-07T07:33:44.768489+0000","last_peered":"2023-12-07T07:33:44.768489+0000","last_clean":"2023-12-07T07:33:44.768489+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:33:44.768489+0000","last_undegraded":"2023-12-07T07:33:44.768489+0000","last_fullsized":"2023-12-07T07:33:44.768489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:11:45.155210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:45.266183+0000","last_change":"2023-12-07T07:33:42.656077+0000","last_active":"2023-12-07T07:33:45.266183+0000","last_peered":"2023-12-07T07:33:45.266183+0000","last_clean":"2023-12-07T07:33:45.266183+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:33:45.266183+0000","last_undegraded":"2023-12-07T07:33:45.266183+0000","last_fullsized":"2023-12-07T07:33:45.266183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:05:05.686340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.759576+0000","last_change":"2023-12-07T07:33:42.617026+0000","last_active":"2023-12-07T07:33:44.759576+0000","last_peered":"2023-12-07T07:33:44.759576+0000","last_clean":"2023-12-07T07:33:44.759576+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:33:44.759576+0000","last_undegraded":"2023-12-07T07:33:44.759576+0000","last_fullsized":"2023-12-07T07:33:44.759576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:58.336948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:45.244239+0000","last_change":"2023-12-07T07:33:42.269533+0000","last_active":"2023-12-07T07:33:45.244239+0000","last_peered":"2023-12-07T07:33:45.244239+0000","last_clean":"2023-12-07T07:33:45.244239+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:33:45.244239+0000","last_undegraded":"2023-12-07T07:33:45.244239+0000","last_fullsized":"2023-12-07T07:33:45.244239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:12:50.238470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.750832+0000","last_change":"2023-12-07T07:33:42.349265+0000","last_active":"2023-12-07T07:33:44.750832+0000","last_peered":"2023-12-07T07:33:44.750832+0000","last_clean":"2023-12-07T07:33:44.750832+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:33:44.750832+0000","last_undegraded":"2023-12-07T07:33:44.750832+0000","last_fullsized":"2023-12-07T07:33:44.750832+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:03:37.867596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:44.762583+0000","last_change":"2023-12-07T07:33:42.649680+0000","last_active":"2023-12-07T07:33:44.762583+0000","last_peered":"2023-12-07T07:33:44.762583+0000","last_clean":"2023-12-07T07:33:44.762583+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:33:44.762583+0000","last_undegraded":"2023-12-07T07:33:44.762583+0000","last_fullsized":"2023-12-07T07:33:44.762583+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:13:55.895630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-07T07:33:46.196187+0000","last_change":"2023-12-07T07:33:36.384672+0000","last_active":"2023-12-07T07:33:46.196187+0000","last_peered":"2023-12-07T07:33:46.196187+0000","last_clean":"2023-12-07T07:33:46.196187+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:33:46.196187+0000","last_undegraded":"2023-12-07T07:33:46.196187+0000","last_fullsized":"2023-12-07T07:33:46.196187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_clean_scrub_stamp":"2023-12-07T07:33:30.679434+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:54:14.180629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444038,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1150000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.056000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.751999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.946000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3079999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.385000000000002}]}]},{"osd":2,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.766}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8550000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.451000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.922999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.901}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.036000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.902999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":634880,"data_stored":483283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.861999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.968}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.183}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.574}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":46.597999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.531000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":52.234999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.808}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.09}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.468}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1649999999999991}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.646000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.428999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.647}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.981}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.411}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":45.107999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":271.90699999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.309000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.968}]}]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4089999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.288}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.384}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.484999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6630000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.097000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.768000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":638976,"data_stored":483656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.106}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.471}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3859999999999992}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.727}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.336}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.903}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2530000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":85816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7729999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6479999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.071}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.002000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.425000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.933}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.75}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:33:51.676 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-07T07:33:51.676 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-07T07:33:51.676 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-07T07:33:51.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.772+0000 7fc141b33640 10 monclient: get_monmap_and_config 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.772+0000 7fc141b33640 10 monclient: build_initial_monmap 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.772+0000 7fc141b33640 10 monclient: monmap: 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:33:51.773152+0000 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:33:51.773152+0000 2023-12-07T07:33:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:33:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:33:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:33:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:33:51.775 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:33:51.775 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.772+0000 7fc141b33640 1 Processor -- start 2023-12-07T07:33:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.772+0000 7fc141b33640 1 -- start start 2023-12-07T07:33:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.772+0000 7fc141b33640 10 monclient: init 2023-12-07T07:33:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-12-07T07:33:51.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc13c0a8910 0x7fc13c13ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): picked mon.noname-c con 0x7fc13c0a8910 addr v2:172.21.15.154:3300/0 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a9b70 0x7fc13c13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): picked mon.noname-b con 0x7fc13c0a9b70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc13c141f90 0x7fc13c142400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): picked mon.noname-a con 0x7fc13c141f90 addr v2:172.21.15.97:3300/0 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c0a8cf0 con 0x7fc13c141f90 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c13f6e0 con 0x7fc13c0a8910 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0aabd0 con 0x7fc13c0a9b70 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.773+0000 7fc141b33640 10 monclient(hunting): authenticate will time out at 1354.151806s 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13affd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a9b70 0x7fc13c13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13bfff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc13c141f90 0x7fc13c142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13affd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a9b70 0x7fc13c13f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:44652/0 (socket says 172.21.15.97:44652) 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13affd640 1 -- 172.21.15.97:0/3562926313 learned_addr learned my addr 172.21.15.97:0/3562926313 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13affd640 10 monclient(hunting): get_auth_request con 0x7fc13c0a9b70 auth_method 0 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13bfff640 10 monclient(hunting): get_auth_request con 0x7fc13c141f90 auth_method 0 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13b7fe640 1 --2- 172.21.15.97:0/3562926313 >> v2:172.21.15.154:3300/0 conn(0x7fc13c0a8910 0x7fc13c13ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13b7fe640 10 monclient(hunting): get_auth_request con 0x7fc13c0a8910 auth_method 0 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.774+0000 7fc13b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.775+0000 7fc13b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 10 monclient(hunting): handle_auth_done global_id 4414 payload 274 2023-12-07T07:33:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 1 -- 172.21.15.97:0/3562926313 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a9b70 msgr2=0x7fc13c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 1 --2- 172.21.15.97:0/3562926313 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a9b70 0x7fc13c13f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 1 -- 172.21.15.97:0/3562926313 >> v2:172.21.15.97:3300/0 conn(0x7fc13c141f90 msgr2=0x7fc13c142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:33:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 1 --2- 172.21.15.97:0/3562926313 >> v2:172.21.15.97:3300/0 conn(0x7fc13c141f90 0x7fc13c142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:33:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 1 monclient: found mon.noname-c 2023-12-07T07:33:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:33:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.777+0000 7fc13b7fe640 1 -- 172.21.15.97:0/3562926313 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0aaeb0 con 0x7fc13c0a8910 2023-12-07T07:33:51.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.779+0000 7fc13b7fe640 1 --2- 172.21.15.97:0/3562926313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c0a8910 0x7fc13c13ec60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0b5310 tx=0x7fc13002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7c964860eab60a0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc13a7fc640 1 -- 172.21.15.97:0/3562926313 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13002e070 con 0x7fc13c0a8910 2023-12-07T07:33:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc13a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc13a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc13a7fc640 10 monclient: dump: 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc13a7fc640 10 monclient: _finish_auth 0 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc13a7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc13a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc141b33640 5 monclient: authenticate success, global_id 4414 2023-12-07T07:33:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.781+0000 7fc141b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:33:51.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc13a7fc640 1 -- 172.21.15.97:0/3562926313 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1300315a0 con 0x7fc13c0a8910 2023-12-07T07:33:51.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc13a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:51.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc13a7fc640 1 -- 172.21.15.97:0/3562926313 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130031930 con 0x7fc13c0a8910 2023-12-07T07:33:51.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc13a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc13a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc13a7fc640 10 monclient: dump: 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc141b33640 10 monclient: get_monmap_and_config success 2023-12-07T07:33:51.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc141b33640 4 set_mon_vals no callback set 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc141b33640 10 monclient: shutdown 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc141b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc141b33640 1 -- 172.21.15.97:0/3562926313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c0a8910 msgr2=0x7fc13c13ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.783+0000 7fc141b33640 1 --2- 172.21.15.97:0/3562926313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c0a8910 0x7fc13c13ec60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0b5310 tx=0x7fc13002b840 comp rx=0 tx=0).stop 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 -- 172.21.15.97:0/3562926313 shutdown_connections 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 --2- 172.21.15.97:0/3562926313 >> v2:172.21.15.97:3300/0 conn(0x7fc13c141f90 0x7fc13c142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 --2- 172.21.15.97:0/3562926313 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a9b70 0x7fc13c13f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 --2- 172.21.15.97:0/3562926313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c0a8910 0x7fc13c13ec60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 -- 172.21.15.97:0/3562926313 >> 172.21.15.97:0/3562926313 conn(0x7fc13c0b7780 msgr2=0x7fc13c0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 -- 172.21.15.97:0/3562926313 shutdown_connections 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 -- 172.21.15.97:0/3562926313 wait complete. 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 10 monclient: build_initial_monmap 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 10 monclient: monmap: 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:33:51.789 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 librados: starting msgr at 2023-12-07T07:33:51.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.784+0000 7fc141b33640 1 librados: starting objecter 2023-12-07T07:33:51.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 Processor -- start 2023-12-07T07:33:51.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 -- start start 2023-12-07T07:33:51.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 librados: setting wanted keys 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 librados: calling monclient init 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient: init 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient: _reopen_session rank -1 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a8910 0x7fc13c11f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): picked mon.noname-c con 0x7fc13c0a8910 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c141f90 0x7fc13c13ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): picked mon.noname-b con 0x7fc13c141f90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc13c0b0f90 0x7fc13c13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): picked mon.noname-a con 0x7fc13c0b0f90 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0ae6b0 con 0x7fc13c0b0f90 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b03c0 con 0x7fc13c141f90 2023-12-07T07:33:51.793 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c11fb10 con 0x7fc13c0a8910 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): _renew_subs 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.785+0000 7fc141b33640 10 monclient(hunting): authenticate will time out at 1354.164075s 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc13c0b0f90 0x7fc13c13b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a8910 0x7fc13c11f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c141f90 0x7fc13c13ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc13c0b0f90 0x7fc13c13b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47968/0 (socket says 172.21.15.97:47968) 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13bfff640 1 -- 172.21.15.97:0/242777962 learned_addr learned my addr 172.21.15.97:0/242777962 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13bfff640 10 monclient(hunting): get_auth_request con 0x7fc13c0b0f90 auth_method 0 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13affd640 10 monclient(hunting): get_auth_request con 0x7fc13c141f90 auth_method 0 2023-12-07T07:33:51.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13b7fe640 10 monclient(hunting): get_auth_request con 0x7fc13c0a8910 auth_method 0 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.786+0000 7fc13b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.787+0000 7fc13b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 10 monclient(hunting): handle_auth_done global_id 4417 payload 824 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 1 -- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a8910 msgr2=0x7fc13c11f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a8910 0x7fc13c11f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 1 -- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc13c0b0f90 msgr2=0x7fc13c13b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc13c0b0f90 0x7fc13c13b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 10 monclient: _finish_hunting 0 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 1 monclient: found mon.noname-b 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:33:51.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.790+0000 7fc13affd640 1 -- 172.21.15.97:0/242777962 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130009400 con 0x7fc13c141f90 2023-12-07T07:33:51.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.792+0000 7fc13affd640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c141f90 0x7fc13c13ace0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e850 tx=0x7fc12c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:51.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.792+0000 7fc123fff640 10 client.?.objecter ms_handle_connect 0x7fc13c141f90 2023-12-07T07:33:51.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.792+0000 7fc123fff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:33:51.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 1 -- 172.21.15.97:0/242777962 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00c940 con 0x7fc13c141f90 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: dump: 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.800 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: _finish_auth 0 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: _check_auth_tickets 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 1 -- 172.21.15.97:0/242777962 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c00caa0 con 0x7fc13c141f90 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 1 -- 172.21.15.97:0/242777962 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00ce10 con 0x7fc13c141f90 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc123fff640 10 monclient: dump: 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:33:51.801 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 5 monclient: authenticate success, global_id 4417 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 10 monclient: _renew_subs 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c13b760 con 0x7fc13c141f90 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc139ffb640 4 set_mon_vals no callback set 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 10 monclient: _renew_subs 2023-12-07T07:33:51.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:51.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c1d7030 con 0x7fc13c141f90 2023-12-07T07:33:51.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.796+0000 7fc141b33640 1 librados: init done 2023-12-07T07:33:51.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.797+0000 7fc141b33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:51.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.797+0000 7fc141b33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:51.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.797+0000 7fc141b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:51.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.797+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c075b60 con 0x7fc13c141f90 2023-12-07T07:33:51.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.816+0000 7fc123fff640 1 -- 172.21.15.97:0/242777962 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc12c013470 con 0x7fc13c141f90 2023-12-07T07:33:51.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.816+0000 7fc123fff640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc12806c820 0x7fc12806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:33:51.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.816+0000 7fc13b7fe640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc12806c820 0x7fc12806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:33:51.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.816+0000 7fc13b7fe640 10 monclient: get_auth_request con 0x7fc12806c820 auth_method 0 2023-12-07T07:33:51.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.816+0000 7fc13b7fe640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc12806c820 0x7fc12806ece0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc130036000 tx=0x7fc13003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:33:51.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:51.817+0000 7fc123fff640 10 client.4417.objecter ms_handle_connect 0x7fc12806c820 2023-12-07T07:33:52.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.047+0000 7fc123fff640 1 -- 172.21.15.97:0/242777962 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc12c058d10 con 0x7fc13c141f90 2023-12-07T07:33:52.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.047+0000 7fc123fff640 10 client.4417.objecter ms_dispatch 0x7fc13c000b90 osd_map(24..24 src has 1..24) v4 2023-12-07T07:33:52.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.047+0000 7fc123fff640 3 client.4417.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:33:52.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.047+0000 7fc123fff640 3 client.4417.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:33:52.050 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.047+0000 7fc123fff640 20 client.4417.objecter dump_active .. 0 homeless 2023-12-07T07:33:52.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.059+0000 7fc123fff640 1 -- 172.21.15.97:0/242777962 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc12c055050 con 0x7fc13c141f90 2023-12-07T07:33:52.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.059+0000 7fc123fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:33:52.061 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.059+0000 7fc123fff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:33:52.268 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.266+0000 7fc141b33640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-12-07T07:33:52.268 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.266+0000 7fc141b33640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-12-07T07:33:52.268 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.266+0000 7fc141b33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:33:52.268 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.266+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc13c0725d0 con 0x7fc13c141f90 2023-12-07T07:33:52.275 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.273+0000 7fc123fff640 1 -- 172.21.15.97:0/242777962 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc12c017080 con 0x7fc13c141f90 2023-12-07T07:33:52.275 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.273+0000 7fc123fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-12-07T07:33:52.275 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.273+0000 7fc123fff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:33:52.275 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:33:52.276 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-07T07:33:52.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 10 librados: watch_flush enter 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 10 librados: watch_flush exit 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 20 client.4417.objecter shutdown clearing up homeless session... 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 10 client.4417.objecter successfully canceled tick 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc12806c820 msgr2=0x7fc12806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc12806c820 0x7fc12806ece0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc130036000 tx=0x7fc13003d040 comp rx=0 tx=0).stop 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 10 monclient: shutdown 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c141f90 msgr2=0x7fc13c13ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c141f90 0x7fc13c13ace0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e850 tx=0x7fc12c00ed10 comp rx=0 tx=0).stop 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 shutdown_connections 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc12806c820 0x7fc12806ece0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc13c0b0f90 0x7fc13c13b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc13c141f90 0x7fc13c13ace0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:52.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 --2- 172.21.15.97:0/242777962 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc13c0a8910 0x7fc13c11f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:33:52.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 >> 172.21.15.97:0/242777962 conn(0x7fc13c0b7780 msgr2=0x7fc13c0b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:33:52.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 shutdown_connections 2023-12-07T07:33:52.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 -- 172.21.15.97:0/242777962 wait complete. 2023-12-07T07:33:52.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:33:52.275+0000 7fc141b33640 1 librados: shutdown 2023-12-07T07:33:52.289 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-07T07:33:52.289 INFO:teuthology.run_tasks:Running task workunit... 2023-12-07T07:33:52.304 INFO:tasks.workunit:Pulling workunits from ref e068ebcf7ecc6503f24666fb6b152034d3fe1067 2023-12-07T07:33:52.305 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-07T07:33:52.305 DEBUG:teuthology.orchestra.run.smithi097:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-07T07:33:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T07:33:52.320 INFO:teuthology.orchestra.run.smithi097.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-07T07:33:52.320 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-07T07:33:52.375 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-07T07:33:52.375 DEBUG:teuthology.orchestra.run.smithi097:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-07T07:33:52.430 INFO:tasks.workunit:timeout=6h 2023-12-07T07:33:52.430 INFO:tasks.workunit:cleanup=True 2023-12-07T07:33:52.431 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e068ebcf7ecc6503f24666fb6b152034d3fe1067 2023-12-07T07:33:52.486 INFO:tasks.workunit.client.0.smithi097.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-07T07:34:03.429 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:29.878 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.431 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:29.880 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.466 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:29.916 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.476 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:29.914 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.481 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:29.919 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.492 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:29.941 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.499 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:29.949 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.517 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:29.955 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:03.536 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:29.984 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.468 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:49.906 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.469 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:49.907 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.480 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:49.929 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.483 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:01:49.921 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.491 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:49.940 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.497 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:49.945 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.499 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:49.948 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.534 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:49.984 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:23.550 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:49.999 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:39.117 INFO:tasks.workunit.client.0.smithi097.stderr:Note: switching to 'e068ebcf7ecc6503f24666fb6b152034d3fe1067'. 2023-12-07T07:34:39.117 INFO:tasks.workunit.client.0.smithi097.stderr: 2023-12-07T07:34:39.117 INFO:tasks.workunit.client.0.smithi097.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr:state without impacting any branches by switching back to a branch. 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: git switch -c 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr:Or undo this operation with: 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: git switch - 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-07T07:34:39.118 INFO:tasks.workunit.client.0.smithi097.stderr: 2023-12-07T07:34:39.119 INFO:tasks.workunit.client.0.smithi097.stderr:HEAD is now at e068ebcf7ec Merge branch 'wip-rf-schedRepair' of https://github.com/ronen-fr/ceph into wip-yuri8-testing-2023-12-06-1425 2023-12-07T07:34:39.122 DEBUG:teuthology.orchestra.run.smithi097:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-07T07:34:39.188 INFO:tasks.workunit.client.0.smithi097.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-07T07:34:39.190 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-07T07:34:39.190 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-07T07:34:39.867 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-07T07:34:39.932 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-07T07:34:39.971 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-07T07:34:39.972 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-07T07:34:39.972 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-07T07:34:40.011 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-07T07:34:40.014 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-07T07:34:40.014 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-07T07:34:40.069 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-12-07T07:34:40.071 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-12-07T07:34:40.071 DEBUG:teuthology.orchestra.run.smithi097:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e068ebcf7ecc6503f24666fb6b152034d3fe1067 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-12-07T07:34:40.128 INFO:tasks.workunit.client.0.smithi097.stderr:+ parallel=1 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' '' = --serial ']' 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ color= 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -t 1 ']' 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ declare -A pids 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_aio 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_aio' 2023-12-07T07:34:40.129 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_aio 2023-12-07T07:34:40.130 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.130 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_aio 2023-12-07T07:34:40.130 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92505 2023-12-07T07:34:40.130 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_aio on pid 92505' 2023-12-07T07:34:40.130 INFO:tasks.workunit.client.0.smithi097.stdout:test api_aio on pid 92505 2023-12-07T07:34:40.133 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92505 2023-12-07T07:34:40.133 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.133 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.133 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-12-07T07:34:40.133 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_aio_pp 2023-12-07T07:34:40.133 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_aio_pp' 2023-12-07T07:34:40.133 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_aio_pp 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.134 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_aio_pp 2023-12-07T07:34:40.135 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92512 2023-12-07T07:34:40.136 INFO:tasks.workunit.client.0.smithi097.stdout:test api_aio_pp on pid 92512 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_aio_pp on pid 92512' 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92512 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_io 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_io' 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_io 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.140 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.141 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.142 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_io 2023-12-07T07:34:40.143 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92522 2023-12-07T07:34:40.144 INFO:tasks.workunit.client.0.smithi097.stdout:test api_io on pid 92522 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_io on pid 92522' 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92522 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_io_pp 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_io_pp' 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_io_pp 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.148 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.149 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.150 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.150 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.150 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.150 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.150 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.151 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.152 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_io_pp 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92533 2023-12-07T07:34:40.153 INFO:tasks.workunit.client.0.smithi097.stdout:test api_io_pp on pid 92533 2023-12-07T07:34:40.159 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_io_pp on pid 92533' 2023-12-07T07:34:40.159 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92533 2023-12-07T07:34:40.159 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.159 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_asio 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_asio' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_asio 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.160 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.161 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.162 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.163 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.164 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_asio 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92547 2023-12-07T07:34:40.165 INFO:tasks.workunit.client.0.smithi097.stdout:test api_asio on pid 92547 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_asio on pid 92547' 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92547 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_list 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_list' 2023-12-07T07:34:40.172 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_list 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.173 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.174 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.175 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.176 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.177 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.178 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92505/exe 2023-12-07T07:34:40.179 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.180 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_list 2023-12-07T07:34:40.180 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92564 2023-12-07T07:34:40.180 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_list on pid 92564' 2023-12-07T07:34:40.180 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-12-07T07:34:40.180 INFO:tasks.workunit.client.0.smithi097.stdout:test api_list on pid 92564 2023-12-07T07:34:40.188 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92564 2023-12-07T07:34:40.188 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.188 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.188 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_lock 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_lock' 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_lock 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.189 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.190 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.191 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92512/exe 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.192 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.193 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_aio: /' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.194 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.195 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_aio.log 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_lock 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.196 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92585 2023-12-07T07:34:40.197 INFO:tasks.workunit.client.0.smithi097.stdout:test api_lock on pid 92585 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_lock on pid 92585' 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92585 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.207 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.208 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_lock_pp 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_lock_pp' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.209 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_aio_pp: /' 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.210 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_lock_pp 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.211 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.212 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.213 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.214 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92522/exe 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.215 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_lock_pp 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92606 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.216 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.217 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.217 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.217 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.217 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.217 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.217 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.217 INFO:tasks.workunit.client.0.smithi097.stdout:test api_lock_pp on pid 92606 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_lock_pp on pid 92606' 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92606 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.227 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_misc 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_misc' 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.228 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.229 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_io.log 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.230 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_io: /' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92533/exe 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.231 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_misc 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.232 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.233 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.234 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.235 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_misc 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92631 2023-12-07T07:34:40.236 INFO:tasks.workunit.client.0.smithi097.stdout:test api_misc on pid 92631 2023-12-07T07:34:40.244 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_misc on pid 92631' 2023-12-07T07:34:40.244 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92631 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_io_pp: /' 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_misc_pp 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_misc_pp' 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.245 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.246 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92547/exe 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.247 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.248 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.249 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.250 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.251 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.252 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_misc_pp 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_misc_pp 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92650 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_misc_pp on pid 92650' 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.253 INFO:tasks.workunit.client.0.smithi097.stdout:test api_misc_pp on pid 92650 2023-12-07T07:34:40.260 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92650 2023-12-07T07:34:40.260 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_asio.log 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_asio: /' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.261 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.262 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.263 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_tier_pp 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_tier_pp' 2023-12-07T07:34:40.264 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.265 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.266 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_tier_pp 2023-12-07T07:34:40.266 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.266 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_tier_pp 2023-12-07T07:34:40.266 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92665 2023-12-07T07:34:40.266 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_tier_pp on pid 92665' 2023-12-07T07:34:40.266 INFO:tasks.workunit.client.0.smithi097.stdout:test api_tier_pp on pid 92665 2023-12-07T07:34:40.274 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92665 2023-12-07T07:34:40.274 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.274 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_pool 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_pool' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.275 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.276 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.277 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.278 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_pool 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.279 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92564/exe 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.280 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.281 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.281 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.281 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.281 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_pool 2023-12-07T07:34:40.281 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92678 2023-12-07T07:34:40.281 INFO:tasks.workunit.client.0.smithi097.stdout:test api_pool on pid 92678 2023-12-07T07:34:40.291 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_pool on pid 92678' 2023-12-07T07:34:40.291 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92678 2023-12-07T07:34:40.291 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.291 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.291 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_snapshots 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.292 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_snapshots' 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.293 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.294 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.297 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.297 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_list.log 2023-12-07T07:34:40.297 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.297 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.297 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.298 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.299 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.300 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_list: /' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92585/exe 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.301 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.302 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_snapshots 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.303 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.304 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.305 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.305 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.305 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.305 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.305 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.305 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_snapshots 2023-12-07T07:34:40.305 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92706 2023-12-07T07:34:40.306 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_snapshots on pid 92706' 2023-12-07T07:34:40.306 INFO:tasks.workunit.client.0.smithi097.stdout:test api_snapshots on pid 92706 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92706 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.311 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92606/exe 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_snapshots_pp 2023-12-07T07:34:40.312 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.313 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_snapshots_pp' 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.314 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_snapshots_pp 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_snapshots_pp 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92720 2023-12-07T07:34:40.315 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_snapshots_pp on pid 92720' 2023-12-07T07:34:40.316 INFO:tasks.workunit.client.0.smithi097.stdout:test api_snapshots_pp on pid 92720 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92720 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.323 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_stat 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_stat' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.324 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92631/exe 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.325 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.326 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_stat 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_stat 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_lock_pp: /' 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92738 2023-12-07T07:34:40.327 INFO:tasks.workunit.client.0.smithi097.stdout:test api_stat on pid 92738 2023-12-07T07:34:40.341 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_stat on pid 92738' 2023-12-07T07:34:40.341 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92738 2023-12-07T07:34:40.341 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.341 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_misc.log 2023-12-07T07:34:40.342 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_misc: /' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.343 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.344 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.345 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.346 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_stat_pp 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_stat_pp' 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_lock: /' 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_lock.log 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_stat_pp 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2023-12-07T07:34:40.347 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.348 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.349 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.350 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.351 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.352 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.353 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.354 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.354 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_stat_pp 2023-12-07T07:34:40.354 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92786 2023-12-07T07:34:40.354 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_stat_pp on pid 92786' 2023-12-07T07:34:40.354 INFO:tasks.workunit.client.0.smithi097.stdout:test api_stat_pp on pid 92786 2023-12-07T07:34:40.372 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92786 2023-12-07T07:34:40.372 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.372 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.372 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.372 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.372 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.372 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92665/exe 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.373 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.374 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.375 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.376 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_watch_notify 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_watch_notify' 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.377 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_watch_notify 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.378 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.379 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.380 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.381 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.382 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_tier_pp: /' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.383 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.384 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.385 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.386 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.387 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_watch_notify 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92855 2023-12-07T07:34:40.388 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_watch_notify on pid 92855' 2023-12-07T07:34:40.389 INFO:tasks.workunit.client.0.smithi097.stdout:test api_watch_notify on pid 92855 2023-12-07T07:34:40.395 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92855 2023-12-07T07:34:40.395 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.395 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.395 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.395 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.395 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92720/exe 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_watch_notify_pp 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_watch_notify_pp' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.396 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92678/exe 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.397 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.398 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_watch_notify_pp 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.399 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-12-07T07:34:40.400 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_watch_notify_pp 2023-12-07T07:34:40.401 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.401 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92874 2023-12-07T07:34:40.401 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_watch_notify_pp on pid 92874' 2023-12-07T07:34:40.401 INFO:tasks.workunit.client.0.smithi097.stdout:test api_watch_notify_pp on pid 92874 2023-12-07T07:34:40.413 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92874 2023-12-07T07:34:40.413 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.413 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_cmd 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_cmd' 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_cmd 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.414 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.415 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_pool.log 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.416 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.417 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92738/exe 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.418 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_pool: /' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.419 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.420 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.421 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_stat.log 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.422 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_cmd 2023-12-07T07:34:40.423 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92919 2023-12-07T07:34:40.424 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_cmd on pid 92919' 2023-12-07T07:34:40.424 INFO:tasks.workunit.client.0.smithi097.stdout:test api_cmd on pid 92919 2023-12-07T07:34:40.425 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92919 2023-12-07T07:34:40.425 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_stat: /' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.426 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.427 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92650/exe 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [----------] Global test environment set-up. 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [----------] 12 tests from AsioRados 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.270+0000 7f4d97e44780 10 monclient: get_monmap_and_config 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.270+0000 7f4d97e44780 10 monclient: build_initial_monmap 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.270+0000 7f4d97e44780 10 monclient: monmap: 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: epoch 0 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: last_changed 0.000000 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: created 0.000000 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-07T07:34:40.428 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: election_strategy: 1 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.270+0000 7f4d97e44780 1 Processor -- start 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.270+0000 7f4d97e44780 1 -- start start 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.270+0000 7f4d97e44780 10 monclient: init 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient: _reopen_session rank -1 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 0x55c1ba325090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): picked mon.noname-a con 0x55c1ba3285f0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 0x55c1ba3247d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): picked mon.noname-b con 0x55c1ba3bdad0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba3ac170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:40.429 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): picked mon.noname-c con 0x55c1ba3c0aa0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): start opening mon connection 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1ba2a1780 con 0x55c1ba3285f0 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): start opening mon connection 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1ba23b970 con 0x55c1ba3bdad0 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): start opening mon connection 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c1ba23b610 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): _renew_subs 2023-12-07T07:34:40.430 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.271+0000 7f4d97e44780 10 monclient(hunting): authenticate will time out at 1402.649896s 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.437 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.438 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92786/exe 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.439 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.440 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.441 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_cmd_pp 2023-12-07T07:34:40.442 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_cmd_pp' 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-12-07T07:34:40.443 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.276+0000 7f4d95936640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba3ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.277+0000 7f4d95936640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba3ac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:60782/0 (socket says 172.21.15.97:60782) 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.277+0000 7f4d95936640 1 -- 172.21.15.97:0/2811311130 learned_addr learned my addr 172.21.15.97:0/2811311130 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.277+0000 7f4d95936640 10 monclient(hunting): get_auth_request con 0x55c1ba3c0aa0 auth_method 0 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.277+0000 7f4d95936640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.277+0000 7f4d95936640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.277+0000 7f4d95936640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.278+0000 7f4d95135640 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 0x55c1ba325090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.279+0000 7f4d95135640 10 monclient(hunting): get_auth_request con 0x55c1ba3285f0 auth_method 0 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.279+0000 7f4d95135640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.279+0000 7f4d95135640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.279+0000 7f4d95135640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:34:40.444 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.282+0000 7f4d95135640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.282+0000 7f4d95135640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.282+0000 7f4d95135640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 1 -- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 msgr2=0x55c1ba3ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba3ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 1 -- 172.21.15.97:0/2811311130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 msgr2=0x55c1ba3247d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 0x55c1ba3247d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 10 monclient: _finish_hunting 0 2023-12-07T07:34:40.445 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 1 monclient: found mon.noname-a 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_stat_pp: /' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_misc_pp: /' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.449 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92706/exe 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_cmd_pp 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.450 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_cmd_pp 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92973 2023-12-07T07:34:40.451 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_cmd_pp on pid 92973' 2023-12-07T07:34:40.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 20 monclient: _un_backoff reopen_interval_mtest api_cmd_pp on pid 92973 2023-12-07T07:34:40.457 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=92973 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_service 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_service' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_service 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.458 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.459 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.460 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.461 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_snapshots: /' 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_service 2023-12-07T07:34:40.462 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93002 2023-12-07T07:34:40.463 INFO:tasks.workunit.client.0.smithi097.stdout:test api_service on pid 93002 2023-12-07T07:34:40.468 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_service on pid 93002' 2023-12-07T07:34:40.468 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93002 2023-12-07T07:34:40.468 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.468 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.468 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_service_pp 2023-12-07T07:34:40.468 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-12-07T07:34:40.468 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_service_pp' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.469 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.470 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.471 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_service_pp 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.472 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.473 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.473 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_service_pp 2023-12-07T07:34:40.473 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93029 2023-12-07T07:34:40.473 INFO:tasks.workunit.client.0.smithi097.stdout:test api_service_pp on pid 93029 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_service_pp on pid 93029' 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93029 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-12-07T07:34:40.484 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.485 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92855/exe 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_c_write_operations 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.486 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_c_write_operations' 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.487 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.488 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.489 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.490 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92874/exe 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.491 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.492 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.493 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_watch_notify: /' 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_c_write_operations 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_c_write_operations 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93068 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_c_write_operations on pid 93068' 2023-12-07T07:34:40.494 INFO:tasks.workunit.client.0.smithi097.stdout:test api_c_write_operations on pid 93068 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93068 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_c_read_operations 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_c_read_operations' 2023-12-07T07:34:40.502 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_c_read_operations 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.503 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.504 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.505 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.506 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.506 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_c_read_operations 2023-12-07T07:34:40.506 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93079 2023-12-07T07:34:40.506 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_c_read_operations on pid 93079' 2023-12-07T07:34:40.506 INFO:tasks.workunit.client.0.smithi097.stdout:test api_c_read_operations on pid 93079 2023-12-07T07:34:40.514 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93079 2023-12-07T07:34:40.514 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.515 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.516 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.517 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.518 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92973/exe 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.519 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.520 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.521 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout:ultipler now 1 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.289+0000 7f4d95135640 1 -- 172.21.15.97:0/2811311130 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c005d20 con 0x55c1ba3285f0 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.294+0000 7f4d95135640 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 0x55c1ba325090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c051000 tx=0x7f4d8c04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68198d532ae88ad server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 1 -- 172.21.15.97:0/2811311130 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c052640 con 0x55c1ba3285f0 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:34:40.522 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: dump: 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: epoch 1 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: election_strategy: 1 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: _finish_auth 0 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: _check_auth_tickets 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 1 -- 172.21.15.97:0/2811311130 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c0527c0 con 0x55c1ba3285f0 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:34:40.523 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 1 -- 172.21.15.97:0/2811311130 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c052b70 con 0x55c1ba3285f0 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d877fe640 10 monclient: dump: 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: epoch 1 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: election_strategy: 1 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.524 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d97e44780 5 monclient: authenticate success, global_id 4407 2023-12-07T07:34:40.525 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d97e44780 10 monclient: get_monmap_and_config success 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.528 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.529 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/92919/exe 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.530 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.360+0000 7f4d97e44780 4 set_mon_vals no callback set 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.361+0000 7f4d97e44780 10 monclient: shutdown 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.361+0000 7f4d97e44780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:34:40.531 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.361+0000 7f4d97e44780 1 -- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 msgr2=0x55c1ba325090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.361+0000 7f4d97e44780 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 0x55c1ba325090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c051000 tx=0x7f4d8c04c000 comp rx=0 tx=0).stop 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 -- 172.21.15.97:0/2811311130 shutdown_connections 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba3ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 0x55c1ba3247d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 --2- 172.21.15.97:0/2811311130 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 0x55c1ba325090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 -- 172.21.15.97:0/2811311130 >> 172.21.15.97:0/2811311130 conn(0x55c1ba32bbc0 msgr2=0x55c1ba32a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 -- 172.21.15.97:0/2811311130 shutdown_connections 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 -- 172.21.15.97:0/2811311130 wait complete. 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 10 monclient: build_initial_monmap 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 10 monclient: monmap: 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: epoch 0 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: last_changed 0.000000 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: created 0.000000 2023-12-07T07:34:40.532 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: election_strategy: 1 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 librados: starting msgr at 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 librados: starting objecter 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.362+0000 7f4d97e44780 1 Processor -- start 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.366+0000 7f4d97e44780 1 -- start start 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.366+0000 7f4d97e44780 1 librados: setting wanted keys 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.366+0000 7f4d97e44780 1 librados: calling monclient init 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.366+0000 7f4d97e44780 10 monclient: init 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient: _reopen_session rank -1 2023-12-07T07:34:40.533 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s api_cls_remote_reads 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' api_cls_remote_reads' 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.538 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo api_cls_remote_reads 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.539 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_cmd.log 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.540 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93029/exe 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=api_cls_remote_reads 2023-12-07T07:34:40.541 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93141 2023-12-07T07:34:40.542 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 0x55c1ba3b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rxtest api_cls_remote_reads on pid 93141 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test api_cls_remote_reads on pid 93141' 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93141 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.546 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-12-07T07:34:40.547 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.548 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s list_parallel 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' list_parallel' 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.549 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.550 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_cmd: /' 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stdout:=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): picked mon.noname-a con 0x55c1ba3285f0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 0x55c1ba45eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): picked mon.noname-b con 0x55c1ba3bdad0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba45f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:40.551 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): picked mon.noname-c con 0x55c1ba3c0aa0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): start opening mon connection 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1ba21b4e0 con 0x55c1ba3285f0 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): start opening mon connection 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1ba3b4730 con 0x55c1ba3bdad0 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): start opening mon connection 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c1ba2ea6e0 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): _renew_subs 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.367+0000 7f4d97e44780 10 monclient(hunting): authenticate will time out at 1402.745451s 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.385+0000 7f4d95936640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba45f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.386+0000 7f4d95936640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba45f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:60892/0 (socket says 172.21.15.97:60892) 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.386+0000 7f4d95936640 1 -- 172.21.15.97:0/3953626917 learned_addr learned my addr 172.21.15.97:0/3953626917 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.386+0000 7f4d95936640 10 monclient(hunting): get_auth_request con 0x55c1ba3c0aa0 auth_method 0 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.386+0000 7f4d95936640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.386+0000 7f4d95936640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:34:40.552 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.386+0000 7f4d95936640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:34:40.553 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.387+0000 7f4d95936640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:34:40.553 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.387+0000 7f4d95936640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:34:40.553 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.387+0000 7f4d95936640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:34:40.554 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.554 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.554 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo list_parallel 2023-12-07T07:34:40.554 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.554 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.554 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=list_parallel 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93181 2023-12-07T07:34:40.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.389+0000 7f4d94934640 1 --2test list_parallel on pid 93181 2023-12-07T07:34:40.558 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test list_parallel on pid 93181' 2023-12-07T07:34:40.558 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93181 2023-12-07T07:34:40.558 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.558 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.558 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-12-07T07:34:40.558 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s open_pools_parallel 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' open_pools_parallel' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.559 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo open_pools_parallel 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=open_pools_parallel 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93203 2023-12-07T07:34:40.560 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test open_pools_parallel on pid 93203' 2023-12-07T07:34:40.561 INFO:tasks.workunit.client.0.smithi097.stdout:test open_pools_parallel on pid 93203 2023-12-07T07:34:40.564 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93203 2023-12-07T07:34:40.564 INFO:tasks.workunit.client.0.smithi097.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ printf %25s delete_pools_parallel 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.565 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:+ r=' delete_pools_parallel' 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.566 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:++ awk '{print $1}' 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:++ echo delete_pools_parallel 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_service_pp: /' 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+ ff=delete_pools_parallel 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.567 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93223 2023-12-07T07:34:40.568 INFO:tasks.workunit.client.0.smithi097.stdout:test delete_pools_parallel on pid 93223 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo 'test delete_pools_parallel on pid 93223' 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ pids[$f]=93223 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ ret=0 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' 1 -eq 1 ']' 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92665 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92665 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.594 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.595 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93002/exe 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.596 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -f /etc/bashrc ']' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:+ . /etc/bashrc 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.597 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -z '' ']' 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ BASHRCSOURCED=Y 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ shopt -q login_shell 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:+++ umask 2023-12-07T07:34:40.598 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' 0022 -eq 0 ']' 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ SHELL=/bin/bash 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.599 INFO:tasks.workunit.client.0.smithi097.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.600 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_service.log 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.601 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_service: /' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.602 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.603 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.604 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'grep=grep --color=auto' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-07T07:34:40.605 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorls.sh 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' '!' -t 0 ']' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:+++ return 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export S_COLORS=auto 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:+++ S_COLORS=auto 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-07T07:34:40.606 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93068/exe 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.607 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.608 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.609 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.610 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-07T07:34:40.611 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.612 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.613 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.614 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.615 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.616 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.617 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/gawk.sh 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93079/exe 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/lang.sh 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup 2023-12-07T07:34:40.618 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout:- 172.21.15.97:0/3953626917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 0x55c1ba45eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.392+0000 7f4d94934640 10 monclient(hunting): get_auth_request con 0x55c1ba3bdad0 auth_method 0 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.392+0000 7f4d94934640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.392+0000 7f4d94934640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.392+0000 7f4d94934640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 1 -- 172.21.15.97:0/3953626917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 msgr2=0x55c1ba45eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1ba3bdad0 0x55c1ba45eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 1 -- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 msgr2=0x55c1ba3b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55c1ba3285f0 0x55c1ba3b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:34:40.619 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 10 monclient: _finish_hunting 0 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 1 monclient: found mon.noname-c 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.393+0000 7f4d95936640 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1ba261760 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.419+0000 7f4d95936640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba45f040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008400 tx=0x7f4d880084f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.419+0000 7f4d867fc640 10 client.?.objecter ms_handle_connect 0x55c1ba3c0aa0 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.419+0000 7f4d867fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8800f050 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: dump: 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: epoch 1 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.620 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.632 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/locale 2023-12-07T07:34:40.632 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93181/exe 2023-12-07T07:34:40.632 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.632 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.632 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.633 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.634 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ list_parallel: /' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.635 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_list_parallel.log 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.636 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.637 INFO:tasks.workunit.client.0.smithi097.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93223/exe 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:++++ export LANG=en_US.UTF-8 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:++++ LANG=en_US.UTF-8 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ LANG=en_US.UTF-8 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ unset LANG_backup config 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n '' ']' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' dumb = linux ']' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.638 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/less.sh 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:++ for i in /etc/profile.d/*.sh 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:++ '[' '' ']' 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:++ . /etc/profile.d/which2.sh 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93203/exe 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.639 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2023-12-07T07:34:40.640 INFO:tasks.workunit.client.0.smithi097.stderr:+++++ readlink /proc/93141/exe 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:++++ basename /usr/bin/bash 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_declare='declare -f' 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+++ which_opt=-f 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export which_declare 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+++ export -f which 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset i 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:++ unset -f pathmunge 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ export PATH 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ unset rc 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2023-12-07T07:34:40.641 INFO:tasks.workunit.client.0.smithi097.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: election_strategy: 1 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: _finish_auth 0 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: _check_auth_tickets 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:40.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d88010040 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d880158a0 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d867fc640 10 monclient: dump: 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: epoch 1 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: min_mon_release 18 (reef) 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: election_strategy: 1 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 5 monclient: authenticate success, global_id 4394 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 10 monclient: _renew_subs 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c1ba247770 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 10 monclient: _renew_subs 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1ba23a920 con 0x55c1ba3c0aa0 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 1 librados: init done 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d97e44780 10 librados: wait_for_osdmap waiting 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.432+0000 7f4d877fe640 4 set_mon_vals no callback set 2023-12-07T07:34:40.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.476+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4d88005020 con 0x55c1ba3c0aa0 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.476+0000 7f4d867fc640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4d7806aa40 0x7f4d7806cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.477+0000 7f4d95135640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4d7806aa40 0x7f4d7806cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.477+0000 7f4d95135640 10 monclient: get_auth_request con 0x7f4d7806aa40 auth_method 0 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.478+0000 7f4d95135640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4d7806aa40 0x7f4d7806cf00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c05d000 tx=0x7f4d8c005a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.478+0000 7f4d867fc640 10 client.4394.objecter ms_handle_connect 0x7f4d7806aa40 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4d880575b0 con 0x55c1ba3c0aa0 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(24..24 src has 1..24) v4 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [24,24] > 0 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding full epoch 24 2023-12-07T07:34:41.795 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d97e44780 10 librados: wait_for_osdmap done waiting 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d97e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.481+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55c1ba29a760 con 0x55c1ba3c0aa0 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.490+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d880531f0 con 0x55c1ba3c0aa0 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.490+0000 7f4d867fc640 10 monclient: handle_get_version_reply finishing 1 version 24 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.490+0000 7f4d86ffd640 10 client.4394.objecter _get_latest_version latest 24, have it 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.490+0000 7f4d97e44780 10 client.4394.objecter create_pool name=ceph_test_rados_api_asio 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.490+0000 7f4d97e44780 10 client.4394.objecter pool_op_submit 1 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.490+0000 7f4d97e44780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:40.490+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55c1ba2ab810 con 0x55c1ba3c0aa0 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v25) v1 ==== 43+0+0 (secure 0 0 0) 0x7f4d88056c80 con 0x55c1ba3c0aa0 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 pool_op_reply(tid 1 (0) Success v25) v1 2023-12-07T07:34:41.796 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 10 client.4394.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v25) v1 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 10 client.4394.objecter have request 1 at 0x55c1ba2e9d30 Op: create 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 20 client.4394.objecter waiting for client to reach epoch 25 before calling back 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 10 monclient: _renew_subs 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=25}) v3 -- 0x7f4d78070780 con 0x55c1ba3c0aa0 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.792+0000 7f4d867fc640 10 client.4394.objecter done 2023-12-07T07:34:42.022 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.900+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== mon.2 v2:172.21.15.97:3301/0 8 ==== osd_map(25..25 src has 1..25) v4 ==== 5836+0+0 (secure 0 0 0) 0x7f4d8805d0d0 con 0x55c1ba3c0aa0 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.900+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(25..25 src has 1..25) v4 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.900+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [25,25] > 24 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.900+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 25 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 10 librados: set snap read head -> 1 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 10 librados: writefull oid=exist nspace= 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba3b53d0 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 25 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x55c1ba45f580 0x55c1ba3a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 2 osd.0 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:42.023 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 2 2023-12-07T07:34:42.024 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 15 client.4394.objecter _send_op 2 to 11.1 on osd.0 2023-12-07T07:34:42.024 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:2 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) v8 -- 0x55c1ba3a58b0 con 0x55c1ba45f580 2023-12-07T07:34:42.024 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:42.024 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:41.901+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:43.901 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:10.350 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:43.911 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:10.360 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:43.981 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:10.418 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:43.983 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:10.421 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:43.985 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:10.434 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:43.992 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:10.441 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:44.020 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:10.469 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:44.039 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:10.477 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:44.074 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:10.523 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:34:48.700 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:42.019+0000 7f4d94934640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x55c1ba45f580 0x55c1ba3a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:34:48.700 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:42.032+0000 7f4d94934640 10 monclient: get_auth_request con 0x55c1ba45f580 auth_method 0 2023-12-07T07:34:48.700 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:42.090+0000 7f4d867fc640 10 client.4394.objecter ms_handle_connect 0x55c1ba45f580 2023-12-07T07:34:48.700 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:42.090+0000 7f4d94934640 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x55c1ba45f580 0x55c1ba3a5370 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:34:48.700 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:43.367+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:48.700 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:43.367+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:48.700 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:43.367+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:43.367+0000 7f4d857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:44.367+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:44.367+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:44.367+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:45.367+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:45.367+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:45.367+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:45.433+0000 7f4d97399640 10 client.4394.objecter tick 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:46.367+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:46.367+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:46.367+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:47.367+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:47.367+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:47.367+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.367+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:48.701 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.367+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:48.702 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.367+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:48.702 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 1 ==== osd_map(26..28 src has 1..28) v4 ==== 14678+0+0 (crc 0 0 0) 0x7f4d90006ce0 con 0x55c1ba45f580 2023-12-07T07:34:48.702 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(26..28 src has 1..28) v4 2023-12-07T07:34:48.702 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [26,28] > 25 2023-12-07T07:34:48.702 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 26 2023-12-07T07:34:48.702 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 10 client.4394.objecter checking op 2 2023-12-07T07:34:48.702 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 26 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:51.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:51.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.697+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 27 2023-12-07T07:34:51.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 10 client.4394.objecter checking op 2 2023-12-07T07:34:51.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 27 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:51.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:51.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 28 2023-12-07T07:34:51.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 10 client.4394.objecter checking op 2 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 28 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:48.698+0000 7f4d867fc640 20 client.4394.objecter 2 11.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:49.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:49.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:49.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:50.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:50.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:50.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:50.434+0000 7f4d97399640 10 client.4394.objecter tick 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:51.948 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 2 ==== osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) v8 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 2 ondisk uv 1 in 11.1 attempt 0 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 2 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 15 client.4394.objecter _finish_op 2 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 2 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:51.949 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d97e44780 10 librados: Objecter returned from writefull r=0 2023-12-07T07:34:52.012 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba3b53d0 2023-12-07T07:34:52.012 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.944+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 28 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.012 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:52.012 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:52.012 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:52.012 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [read 0~256] tid 3 osd.0 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 3 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 15 client.4394.objecter _send_op 3 to 11.1 on osd.0 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:3 11.1 11:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55c1ba462000 con 0x55c1ba45f580 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba4623e0 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 28 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid noexist '@11' '@11' [read 0~256] tid 4 osd.0 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.013 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 4 2023-12-07T07:34:52.014 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 15 client.4394.objecter _send_op 4 to 11.1 on osd.0 2023-12-07T07:34:52.014 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:4 11.1 11:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55c1ba462880 con 0x55c1ba45f580 2023-12-07T07:34:52.014 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:52.014 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:51.945+0000 7f4d97e44780 5 client.4394.objecter 2 in flight 2023-12-07T07:34:52.014 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 3 ==== osd_map(29..30 src has 1..30) v4 ==== 1968+0+0 (crc 0 0 0) 0x7f4d90012330 con 0x55c1ba45f580 2023-12-07T07:34:52.014 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(29..30 src has 1..30) v4 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [29,30] > 28 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 29 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 10 client.4394.objecter checking op 3 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 29 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 10 client.4394.objecter checking op 4 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 29 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:52.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 30 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 10 client.4394.objecter checking op 3 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 30 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 10 client.4394.objecter checking op 4 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 30 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter 3 11.d3a59201 osd.0 exist [read 0~256] 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.009+0000 7f4d867fc640 20 client.4394.objecter 4 11.a7bbaa1 osd.0 noexist [read 0~256] 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 3 ondisk uv 1 in 11.1 attempt 0 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 5 2023-12-07T07:34:52.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 3 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 15 client.4394.objecter _finish_op 3 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 3 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 5 client.4394.objecter 1 in flight 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 4 ondisk uv 0 in 11.1 attempt 0 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 10 client.4394.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 4 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 15 client.4394.objecter _finish_op 4 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 4 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (125 ms) 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba4623e0 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 30 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [read 0~256] tid 5 osd.0 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 5 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 15 client.4394.objecter _send_op 5 to 11.1 on osd.0 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:5 11.1 11:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c1ba462c60 con 0x55c1ba45f580 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:52.077 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba3b53d0 2023-12-07T07:34:52.078 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 30 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.078 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:52.078 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:52.078 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid noexist '@11' '@11' [read 0~256] tid 6 osd.0 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 6 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 15 client.4394.objecter _send_op 6 to 11.1 on osd.0 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:6 11.1 11:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c1ba46b5c0 con 0x55c1ba45f580 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.070+0000 7f4d97e44780 5 client.4394.objecter 2 in flight 2023-12-07T07:34:52.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 5 ondisk uv 1 in 11.1 attempt 0 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 5 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 5 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 15 client.4394.objecter _finish_op 5 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 5 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 5 client.4394.objecter 1 in flight 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 6 ondisk uv 0 in 11.1 attempt 0 2023-12-07T07:34:52.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 10 client.4394.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T07:34:52.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:52.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 6 2023-12-07T07:34:52.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 15 client.4394.objecter _finish_op 6 2023-12-07T07:34:52.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 6 2023-12-07T07:34:52.182 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:52.182 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (30 ms) 2023-12-07T07:34:52.182 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-12-07T07:34:52.182 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba3b53d0 2023-12-07T07:34:52.182 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 30 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.182 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [read 0~256] tid 7 osd.0 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 7 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 15 client.4394.objecter _send_op 7 to 11.1 on osd.0 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:7 11.1 11:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c1ba4623e0 con 0x55c1ba45f580 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba4abfb0 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 30 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:52.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid noexist '@11' '@11' [read 0~256] tid 8 osd.0 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 8 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 15 client.4394.objecter _send_op 8 to 11.1 on osd.0 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:8 11.1 11:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c1ba4ac450 con 0x55c1ba45f580 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.100+0000 7f4d97e44780 5 client.4394.objecter 2 in flight 2023-12-07T07:34:52.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 8 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 7 ondisk uv 1 in 11.1 attempt 0 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 5 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 7 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 15 client.4394.objecter _finish_op 7 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 7 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.179+0000 7f4d94934640 5 client.4394.objecter 1 in flight 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 9 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:52.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 8 ondisk uv 0 in 11.1 attempt 0 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 10 client.4394.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 8 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 15 client.4394.objecter _finish_op 8 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 8 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (105 ms) 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 librados: queue_aio_write 0x55c1ba29a480 completion 0x55c1ba3b53d0 write_seq 1 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba46c930 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 30 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:52.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:52.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [write 0~5 in=5b] tid 9 osd.0 2023-12-07T07:34:53.569 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 9 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 15 client.4394.objecter _send_op 9 to 11.1 on osd.0 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:9 11.1 11:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c1ba3b54e0 con 0x55c1ba45f580 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.205+0000 7f4d97e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 10 ==== osd_map(31..31 src has 1..31) v4 ==== 1007+0+0 (crc 0 0 0) 0x7f4d90001e30 con 0x55c1ba45f580 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(31..31 src has 1..31) v4 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [31,31] > 30 2023-12-07T07:34:53.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 31 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 10 client.4394.objecter checking op 9 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 31 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:52.415+0000 7f4d867fc640 20 client.4394.objecter 9 11.d3a59201 osd.0 exist [write 0~5 in=5b] 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.369+0000 7f4d857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 11 ==== osd_op_reply(9 exist [write 0~5] v31'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(9 exist [write 0~5] v31'2 uv2 ondisk = 0) v8 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 9 ondisk uv 2 in 11.1 attempt 0 2023-12-07T07:34:53.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 9 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 15 client.4394.objecter _finish_op 9 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 9 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d94934640 20 librados: complete_aio_write 0x55c1ba3b53d0 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1362 ms) 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-12-07T07:34:53.650 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-07T07:34:53.651 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 librados: queue_aio_write 0x55c1ba29a480 completion 0x55c1ba46cdd0 write_seq 2 2023-12-07T07:34:53.651 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba46c930 2023-12-07T07:34:53.651 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 31 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:53.651 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:53.651 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:53.651 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:53.651 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [write 0~5 in=5b] tid 10 osd.0 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 10 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 15 client.4394.objecter _send_op 10 to 11.1 on osd.0 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:10 11.1 11:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) v8 -- 0x55c1ba3b54e0 con 0x55c1ba45f580 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.567+0000 7f4d97e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.647+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 12 ==== osd_map(32..32 src has 1..33) v4 ==== 984+0+0 (crc 0 0 0) 0x7f4d90001ff0 con 0x55c1ba45f580 2023-12-07T07:34:53.653 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.647+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(32..32 src has 1..33) v4 2023-12-07T07:34:53.654 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.647+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [32,32] > 31 2023-12-07T07:34:53.654 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.647+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 32 2023-12-07T07:34:53.654 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.648+0000 7f4d867fc640 10 client.4394.objecter checking op 10 2023-12-07T07:34:53.654 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.648+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 32 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:53.654 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.648+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.648+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:53.648+0000 7f4d867fc640 20 client.4394.objecter 10 11.d3a59201 osd.0 exist [write 0~5 in=5b] 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 13 ==== osd_op_reply(10 exist [write 0~5] v32'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(10 exist [write 0~5] v32'3 uv3 ondisk = 0) v8 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:54.768 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 10 ondisk uv 3 in 11.1 attempt 0 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 10 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 15 client.4394.objecter _finish_op 10 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 10 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d94934640 20 librados: complete_aio_write 0x55c1ba46cdd0 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (1198 ms) 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 librados: queue_aio_write 0x55c1ba29a480 completion 0x55c1ba46c930 write_seq 3 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba48f640 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 32 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:54.769 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:54.770 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:54.770 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [write 0~5 in=5b] tid 11 osd.0 2023-12-07T07:34:54.770 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:54.770 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 11 2023-12-07T07:34:54.770 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 15 client.4394.objecter _send_op 11 to 11.1 on osd.0 2023-12-07T07:34:54.770 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:11 11.1 11:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c1ba48fae0 con 0x55c1ba45f580 2023-12-07T07:34:55.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:55.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:55.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.765+0000 7f4d97e44780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-07T07:34:55.208 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 14 ==== osd_map(33..33 src has 1..33) v4 ==== 312+0+0 (crc 0 0 0) 0x7f4d90001b80 con 0x55c1ba45f580 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(33..33 src has 1..33) v4 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [33,33] > 32 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 33 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 10 client.4394.objecter checking op 11 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 33 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:54.821+0000 7f4d867fc640 20 client.4394.objecter 11 11.d3a59201 osd.0 exist [write 0~5 in=5b] 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 15 ==== osd_op_reply(11 exist [write 0~5] v33'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f4d90001760 con 0x55c1ba45f580 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(11 exist [write 0~5] v33'4 uv4 ondisk = 0) v8 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 11 ondisk uv 4 in 11.1 attempt 0 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 11 2023-12-07T07:34:55.209 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 15 client.4394.objecter _finish_op 11 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 11 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d94934640 20 librados: complete_aio_write 0x55c1ba46c930 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (441 ms) 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba46b920 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 33 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:55.210 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:56.842 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [read 0~0] tid 12 osd.0 2023-12-07T07:34:56.842 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:56.842 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 12 2023-12-07T07:34:56.842 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 15 client.4394.objecter _send_op 12 to 11.1 on osd.0 2023-12-07T07:34:56.842 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:12 11.1 11:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55c1ba46bdc0 con 0x55c1ba45f580 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba46c230 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 33 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid noexist '@11' '@11' [read 0~0] tid 13 osd.0 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 13 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 15 client.4394.objecter _send_op 13 to 11.1 on osd.0 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:13 11.1 11:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55c1ba48fef0 con 0x55c1ba45f580 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:56.843 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.206+0000 7f4d97e44780 5 client.4394.objecter 2 in flight 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:55.435+0000 7f4d97399640 10 client.4394.objecter tick 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1633 ms) 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 17 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 12 ondisk uv 4 in 11.1 attempt 0 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 5 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 12 2023-12-07T07:34:56.844 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 15 client.4394.objecter _finish_op 12 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 12 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 5 client.4394.objecter 1 in flight 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 18 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 13 ondisk uv 0 in 11.1 attempt 0 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 10 client.4394.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 13 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 15 client.4394.objecter _finish_op 13 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 13 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:56.845 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 16 ==== osd_map(34..35 src has 1..35) v4 ==== 1275+0+0 (crc 0 0 0) 0x7f4d90012a50 con 0x55c1ba45f580 2023-12-07T07:34:56.846 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(34..35 src has 1..35) v4 2023-12-07T07:34:56.846 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [34,35] > 33 2023-12-07T07:34:56.846 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 34 2023-12-07T07:34:56.846 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 35 2023-12-07T07:34:56.846 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:56.846 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba46c230 2023-12-07T07:34:56.846 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 35 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:57.072 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:57.072 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:57.072 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:57.072 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [read 0~0] tid 14 osd.0 2023-12-07T07:34:57.072 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 14 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 15 client.4394.objecter _send_op 14 to 11.1 on osd.0 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:14 11.1 11:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x55c1ba46b920 con 0x55c1ba45f580 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba48fef0 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 35 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid noexist '@11' '@11' [read 0~0] tid 15 osd.0 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 15 2023-12-07T07:34:57.073 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 15 client.4394.objecter _send_op 15 to 11.1 on osd.0 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:15 11.1 11:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x55c1ba46bdc0 con 0x55c1ba45f580 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:56.839+0000 7f4d97e44780 5 client.4394.objecter 2 in flight 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 19 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 14 ondisk uv 4 in 11.1 attempt 0 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 5 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 14 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 15 client.4394.objecter _finish_op 14 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 14 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 5 client.4394.objecter 1 in flight 2023-12-07T07:34:57.074 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.069+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 20 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 15 ondisk uv 0 in 11.1 attempt 0 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 10 client.4394.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 15 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 15 client.4394.objecter _finish_op 15 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 15 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (231 ms) 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba48efd0 2023-12-07T07:34:57.075 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 35 base exist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [read 0~0] tid 16 osd.0 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 16 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 15 client.4394.objecter _send_op 16 to 11.1 on osd.0 2023-12-07T07:34:57.076 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:16 11.1 11:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x55c1ba46bdc0 con 0x55c1ba45f580 2023-12-07T07:34:57.095 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.095 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:57.095 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba48f470 2023-12-07T07:34:57.095 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 35 base noexist @11 precalc_pgid 0 pgid 0.0 is_read 2023-12-07T07:34:57.095 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter _calc_target target noexist @11 -> pgid 11.a7bbaa1 2023-12-07T07:34:57.095 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.a7bbaa1 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:57.095 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid noexist '@11' '@11' [read 0~0] tid 17 osd.0 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 17 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 15 client.4394.objecter _send_op 17 to 11.1 on osd.0 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:17 11.1 11:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v8 -- 0x55c1ba48f910 con 0x55c1ba45f580 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.070+0000 7f4d97e44780 5 client.4394.objecter 2 in flight 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (22 ms) 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 21 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 16 ondisk uv 4 in 11.1 attempt 0 2023-12-07T07:34:57.096 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 5 2023-12-07T07:34:57.097 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 16 2023-12-07T07:34:57.097 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 15 client.4394.objecter _finish_op 16 2023-12-07T07:34:57.097 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 5 2023-12-07T07:34:57.097 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 16 2023-12-07T07:34:57.097 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 5 client.4394.objecter 1 in flight 2023-12-07T07:34:57.097 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 22 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:57.097 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 17 ondisk uv 0 in 11.1 attempt 0 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 10 client.4394.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 17 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 15 client.4394.objecter _finish_op 17 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 17 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:57.945 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 20 librados: queue_aio_write 0x55c1ba29a480 completion 0x55c1ba46ba30 write_seq 4 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba46c230 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 35 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 18 osd.0 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 18 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 15 client.4394.objecter _send_op 18 to 11.1 on osd.0 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:18 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35) v8 -- 0x55c1ba46bdc0 con 0x55c1ba45f580 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.092+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:57.946 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:57.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 23 ==== osd_op_reply(18 exist [writefull 0~5] v35'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:57.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(18 exist [writefull 0~5] v35'5 uv5 ondisk = 0) v8 2023-12-07T07:34:57.947 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:58.002 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 18 ondisk uv 5 in 11.1 attempt 0 2023-12-07T07:34:58.002 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 18 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 15 client.4394.objecter _finish_op 18 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 18 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d94934640 20 librados: complete_aio_write 0x55c1ba46ba30 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (850 ms) 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 20 librados: queue_aio_write 0x55c1ba29a480 completion 0x55c1ba48fcf0 write_seq 5 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba46c230 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 35 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 19 osd.0 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 19 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 15 client.4394.objecter _send_op 19 to 11.1 on osd.0 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:19 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35) v8 -- 0x55c1ba46bdc0 con 0x55c1ba45f580 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:57.942+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 24 ==== osd_map(36..36 src has 1..36) v4 ==== 304+0+0 (crc 0 0 0) 0x7f4d900124f0 con 0x55c1ba45f580 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(36..36 src has 1..36) v4 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [36,36] > 35 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 36 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 10 client.4394.objecter checking op 19 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 36 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:59.453 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:34:59.453 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.000+0000 7f4d867fc640 20 client.4394.objecter 19 11.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:58.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.369+0000 7f4d857fa640 10 monclient: tick 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.369+0000 7f4d857fa640 10 monclient: _check_auth_tickets 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.369+0000 7f4d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 25 ==== osd_op_reply(19 exist [writefull 0~5] v36'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(19 exist [writefull 0~5] v36'6 uv6 ondisk = 0) v8 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 19 ondisk uv 6 in 11.1 attempt 0 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 19 2023-12-07T07:34:59.454 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 15 client.4394.objecter _finish_op 19 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 19 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d94934640 20 librados: complete_aio_write 0x55c1ba48fcf0 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1509 ms) 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 20 librados: queue_aio_write 0x55c1ba29a480 completion 0x55c1ba46ba30 write_seq 6 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 10 client.4394.objecter _op_submit op 0x55c1ba48efd0 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 20 client.4394.objecter _calc_target epoch 36 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 10 client.4394.objecter _calc_target raw pgid 11.d3a59201 -> actual 11.1 acting [0,1] primary 0 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 20 client.4394.objecter _get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 10 client.4394.objecter _op_submit oid exist '@11' '@11' [writefull 0~5 in=5b] tid 20 osd.0 2023-12-07T07:34:59.455 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 20 client.4394.objecter get_session s=0x55c1ba25f140 osd=0 3 2023-12-07T07:34:59.456 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 15 client.4394.objecter _session_op_assign 0 20 2023-12-07T07:34:59.456 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 15 client.4394.objecter _send_op 20 to 11.1 on osd.0 2023-12-07T07:35:00.302 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- osd_op(unknown.0.0:20 11.1 11:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x55c1ba46bdc0 con 0x55c1ba45f580 2023-12-07T07:35:00.302 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.451+0000 7f4d97e44780 5 client.4394.objecter 1 in flight 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.472+0000 7f4d867fc640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 26 ==== osd_map(37..38 src has 1..38) v4 ==== 1283+0+0 (crc 0 0 0) 0x7f4d90012a60 con 0x55c1ba45f580 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.472+0000 7f4d867fc640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_map(37..38 src has 1..38) v4 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.472+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map got epochs [37,38] > 36 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.472+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 37 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 10 client.4394.objecter checking op 20 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 37 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 3 client.4394.objecter handle_osd_map decoding incremental epoch 38 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 10 client.4394.objecter checking op 20 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 20 client.4394.objecter _calc_target epoch 38 base exist @11 precalc_pgid 0 pgid 0.0 is_write 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 20 client.4394.objecter _calc_target target exist @11 -> pgid 11.d3a59201 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 20 client.4394.objecter dump_active .. 0 homeless 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:34:59.473+0000 7f4d867fc640 20 client.4394.objecter 20 11.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2023-12-07T07:35:00.303 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 1 -- 172.21.15.97:0/3953626917 <== osd.0 v2:172.21.15.97:6818/2798215727 27 ==== osd_op_reply(20 exist [writefull 0~5] v38'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f4d9001d040 con 0x55c1ba45f580 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 10 client.4394.objecter ms_dispatch 0x55c1ba2941b0 osd_op_reply(20 exist [writefull 0~5] v38'7 uv7 ondisk = 0) v8 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 10 client.4394.objecter in handle_osd_op_reply 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 7 client.4394.objecter handle_osd_op_reply 20 ondisk uv 7 in 11.1 attempt 0 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 10 client.4394.objecter op 0 rval 0 len 0 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 15 client.4394.objecter handle_osd_op_reply completed tid 20 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 15 client.4394.objecter _finish_op 20 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 4 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 15 client.4394.objecter _session_op_remove 0 20 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 5 client.4394.objecter 0 in flight 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d94934640 20 librados: complete_aio_write 0x55c1ba46ba30 2023-12-07T07:35:00.304 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (849 ms) 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 10 librados: watch_flush enter 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 10 librados: watch_flush exit 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 10 client.4394.objecter close_session for osd.0 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x55c1ba45f580 msgr2=0x55c1ba3a5370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x55c1ba45f580 0x55c1ba3a5370 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 20 client.4394.objecter put_session s=0x55c1ba25f140 osd=0 1 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 20 client.4394.objecter shutdown clearing up homeless session... 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 10 client.4394.objecter successfully canceled tick 2023-12-07T07:35:00.307 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4d7806aa40 msgr2=0x7f4d7806cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4d7806aa40 0x7f4d7806cf00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c05d000 tx=0x7f4d8c005a00 comp rx=0 tx=0).stop 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 10 monclient: shutdown 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 msgr2=0x55c1ba45f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 --2- 172.21.15.97:0/3953626917 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55c1ba3c0aa0 0x55c1ba45f040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4d88008400 tx=0x7f4d880084f0 comp rx=0 tx=0).stop 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d95936640 1 -- 172.21.15.97:0/3953626917 reap_dead start 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 shutdown_connections 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.300+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 >> 172.21.15.97:0/3953626917 conn(0x55c1ba32bbc0 msgr2=0x55c1ba3b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.301+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 shutdown_connections 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.301+0000 7f4d97e44780 1 -- 172.21.15.97:0/3953626917 wait complete. 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.301+0000 7f4d97e44780 1 librados: shutdown 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [----------] 12 tests from AsioRados (8355 ms total) 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [----------] Global test environment tear-down 2023-12-07T07:35:00.308 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (20031 ms total) 2023-12-07T07:35:00.309 INFO:tasks.workunit.client.0.smithi097.stdout: api_asio: [ PASSED ] 12 tests. 2023-12-07T07:35:03.541 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:29.990 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.614 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:30.063 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.647 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:30.096 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.669 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:30.107 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.671 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:30.109 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.715 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:30.164 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.834 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:30.272 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.880 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:30.329 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:03.906 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:30.355 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:09.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2023-12-07T07:35:09.758 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [----------] Global test environment set-up. 2023-12-07T07:35:09.758 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [----------] 4 tests from LibRadosService 2023-12-07T07:35:09.758 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-12-07T07:35:09.758 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5448 ms) 2023-12-07T07:35:09.758 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ OK ] LibRadosService.RegisterLate (72 ms) 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: cluster: 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: id: f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: health: HEALTH_WARN 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 10 pool(s) do not have an application enabled 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: services: 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: mgr: y(active, since 116s), standbys: x 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: osd: 8 osds: 8 up (since 73s), 8 in (since 116s) 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-12-07T07:35:09.759 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: data: 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: pools: 26 pools, 777 pgs 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: pgs: 84.299% pgs unknown 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 14.543% pgs not active 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 655 unknown 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 113 creating+peering 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 9 active+clean 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: cluster: 2023-12-07T07:35:09.760 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: id: f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: health: HEALTH_WARN 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 7 pool(s) do not have an application enabled 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: services: 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: mgr: y(active, since 118s), standbys: x 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 118s) 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: data: 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: pools: 29 pools, 849 pgs 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-07T07:35:09.761 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: pgs: 48.410% pgs unknown 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 50.530% pgs not active 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 411 unknown 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 319 creating+peering 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 110 creating+activating 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 9 active+clean 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3721 ms) 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ RUN ] LibRadosService.Status 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ OK ] LibRadosService.Status (20056 ms) 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [----------] 4 tests from LibRadosService (29297 ms total) 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [----------] Global test environment tear-down 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29298 ms total) 2023-12-07T07:35:09.762 INFO:tasks.workunit.client.0.smithi097.stdout: api_service: [ PASSED ] 4 tests. 2023-12-07T07:35:11.336 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-12-07T07:35:11.336 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [----------] Global test environment set-up. 2023-12-07T07:35:11.336 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-12-07T07:35:11.336 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-12-07T07:35:11.336 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5450 ms) 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (114 ms) 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20144 ms) 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: attempt 0 of 20 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5181 ms) 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30889 ms total) 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [----------] Global test environment tear-down 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30889 ms total) 2023-12-07T07:35:11.337 INFO:tasks.workunit.client.0.smithi097.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] Global test environment set-up. 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2023-12-07T07:34:40.632+0000 7fddef057780 -1 failed for service _ceph-mon._tcp 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: ceph-mon 2023-12-07T07:35:11.507 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: 2023-12-07T07:34:40.632+0000 7fddef057780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (385 ms) 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2008 ms) 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2393 ms total) 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f289570 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f026d40 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f23f930 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f240200 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f191200 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f3b6680 2023-12-07T07:35:11.508 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f31a510 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f3b79a0 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f3b83a0 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f3b89f0 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f3b9070 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc5f410 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3f3b8140 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc602a0 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc5f180 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc61170 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc60080 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc61ff0 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc60f40 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc62f20 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 0x564a3fc61df0 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: started 21 aios 2023-12-07T07:35:11.509 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f289570 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f026d40 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f23f930 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f240200 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f191200 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f3b6680 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f31a510 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f3b79a0 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f3b83a0 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f3b89f0 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f3b9070 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc5f410 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3f3b8140 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc602a0 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc5f180 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc61170 2023-12-07T07:35:11.510 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc60080 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc61ff0 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc60f40 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc62f20 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: waiting 0x564a3fc61df0 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: done. 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (26621 ms) 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (26621 ms total) 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-12-07T07:35:11.511 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-12-07T07:35:14.235 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:35:14.156+0000 12257640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 10.1c 10.b90c249c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:15.211 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:35:15.203+0000 12257640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 10.1c 10.b90c249c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:15.338 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:35:15.324+0000 12a58640 -1 osd.7 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 19.1a 19.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:15.515 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-12-07T07:35:15.515 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (35000 ms) 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (35000 ms total) 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (35000 ms total) 2023-12-07T07:35:15.516 INFO:tasks.workunit.client.0.smithi097.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-12-07T07:35:16.445 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:35:16.388+0000 12a58640 -1 osd.7 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 19.1a 19.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:16.871 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:35:16.843+0000 12257640 -1 osd.2 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 10.1c 10.b90c249c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:17.510 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:35:17.502+0000 12a58640 -1 osd.7 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 19.1a 19.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:18.371 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:35:18.341+0000 12257640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 10.1c 10:3924309d:test-rados-api-smithi097-93048-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:18.716 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:35:18.662+0000 12a58640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 19.1a 19.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:19.378 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:35:19.333+0000 12257640 -1 osd.2 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 10.1c 10:3924309d:test-rados-api-smithi097-93048-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:20.196 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:35:20.159+0000 12a58640 -1 osd.7 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~161 in=161b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:20.459 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:35:20.362+0000 12257640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 10.1c 10:3924309d:test-rados-api-smithi097-93048-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:21.329 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:35:21.283+0000 12a58640 -1 osd.7 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~161 in=161b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:21.428 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:35:21.423+0000 12257640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 10.1c 10:3924309d:test-rados-api-smithi097-93048-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:22.335 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:35:22.274+0000 12a58640 -1 osd.7 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~161 in=161b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2023-12-07T07:35:24.326 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:50.775 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.386 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:50.835 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.393 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:50.831 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.398 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:50.836 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.450 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:50.899 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.460 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:50.910 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.503 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:50.952 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.506 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:02:50.943 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:24.579 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:51.028 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: Running main() from gmock_main.cc 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [----------] Global test environment set-up. 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolList (4310 ms) 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3881 ms) 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3591 ms) 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3653 ms) 2023-12-07T07:35:26.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3489 ms) 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6648 ms) 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5643 ms) 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6114 ms) 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9203 ms) 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [----------] 9 tests from LibRadosPools (46534 ms total) 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [----------] Global test environment tear-down 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (46534 ms total) 2023-12-07T07:35:26.826 INFO:tasks.workunit.client.0.smithi097.stdout: api_pool: [ PASSED ] 9 tests. 2023-12-07T07:35:44.614 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:11.062 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.629 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:11.078 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.687 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:11.137 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.691 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:11.141 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.693 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:03:11.131 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.697 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:03:11.135 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.759 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:11.208 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.796 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:03:11.234 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:44.796 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:11.245 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (886 ms) 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (114 ms) 2023-12-07T07:35:52.059 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-12-07T07:35:52.060 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (70675 ms) 2023-12-07T07:35:52.060 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (71675 ms total) 2023-12-07T07:35:52.060 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: 2023-12-07T07:35:52.060 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-12-07T07:35:52.060 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (71675 ms total) 2023-12-07T07:35:52.060 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-12-07T07:36:08.633 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: notify 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (22140 ms) 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (10185 ms) 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (32325 ms total) 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: 2023-12-07T07:36:08.634 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: notify 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (5694 ms) 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: notify 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (17937 ms) 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (4571 ms) 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (10543 ms) 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227121805824 notify_id 403726925825 notifier_gid 4720 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (2999 ms) 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227121805824 notify_id 403726925825 notifier_gid 4720 2023-12-07T07:36:08.635 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (687 ms) 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227121805824 notify_id 403726925826 notifier_gid 4720 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (248 ms) 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227123513360 notify_id 403726925825 notifier_gid 4720 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (343 ms) 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227124171568 notify_id 403726925827 notifier_gid 4720 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (264 ms) 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227124171568 notify_id 403726925828 notifier_gid 4720 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (197 ms) 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: trying... 2023-12-07T07:36:08.636 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227124171568 notify_id 408021893123 notifier_gid 4720 2023-12-07T07:36:08.637 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: timed out 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: fl api_c_read_operations: Running main() from gmock_main.cc 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (16354 ms) 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-12-07T07:36:14.101 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (35074 ms) 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (11262 ms) 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (5520 ms) 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (10570 ms) 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (164 ms) 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (298 ms) 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (252 ms) 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-12-07T07:36:14.102 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (153 ms) 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (338 ms) 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (150 ms) 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (102 ms) 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (549 ms) 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (428 ms) 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (314 ms) 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-12-07T07:36:14.103 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (156 ms) 2023-12-07T07:36:14.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (81684 ms total) 2023-12-07T07:36:14.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: 2023-12-07T07:36:14.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-12-07T07:36:14.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (93640 ms total) 2023-12-07T07:36:14.104 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-12-07T07:36:21.938 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-12-07T07:36:21.938 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] Global test environment set-up. 2023-12-07T07:36:21.938 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] 7 tests from LibRadosList 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosList.ListObjects (78728 ms) 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1541 ms) 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset foo1,foo2,foo3 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo1 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo2 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo3 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset foo1,foo4,foo5 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo4 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo5 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo1 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset foo6,foo7 2023-12-07T07:36:21.939 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo7 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo6 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns1:foo4 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns1:foo5 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns2:foo7 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns2:foo6 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns1:foo1 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: :foo1 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: :foo2 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: :foo3 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (4552 ms) 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 1 0 2023-12-07T07:36:21.940 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 10 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 13 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 7 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 14 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 0 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 15 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 11 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 5 0 2023-12-07T07:36:21.941 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 8 0 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 6 0 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 3 0 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 4 0 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 12 0 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 9 0 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 2 0 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (8568 ms) 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: x cursor=MIN 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=1 cursor=18:02547ec2:::1:head 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=10 cursor=18:52ea6a34:::10:head 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=13 cursor=18:566253c9:::13:head 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=7 cursor=18:5c6b0b28:::7:head 2023-12-07T07:36:21.942 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=14 cursor=18:62a1935d:::14:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=0 cursor=18:6cac518f:::0:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=15 cursor=18:863748b0:::15:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=11 cursor=18:89d3ae78:::11:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=5 cursor=18:b29083e3:::5:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=8 cursor=18:bd63b0f1:::8:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=6 cursor=18:c4fdafeb:::6:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=3 cursor=18:cfc208b3:::3:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=4 cursor=18:d83876eb:::4:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=12 cursor=18:de5d7c5f:::12:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=9 cursor=18:e960b815:::9:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > oid=2 cursor=18:f905c69b:::2:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: FIRST> seek to MIN oid=1 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=1 cursor=18:02547ec2:::1:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:02547ec2:::1:head 2023-12-07T07:36:21.943 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:02547ec2:::1:head -> 1 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=10 cursor=18:52ea6a34:::10:head 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:52ea6a34:::10:head 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:52ea6a34:::10:head -> 10 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=13 cursor=18:566253c9:::13:head 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:566253c9:::13:head 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:566253c9:::13:head -> 13 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=7 cursor=18:5c6b0b28:::7:head 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:5c6b0b28:::7:head 2023-12-07T07:36:21.944 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:5c6b0b28:::7:head -> 7 2023-12-07T07:36:23.080 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:49.529 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:23.101 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:49.550 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:23.323 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:49.772 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:23.749 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=14 cursor=18:62a1935d:::14:head 2023-12-07T07:36:23.749 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:62a1935d:::14:head 2023-12-07T07:36:23.749 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:62a1935d:::14:head -> 14 2023-12-07T07:36:23.750 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=0 cursor=18:6cac518f:::0:head 2023-12-07T07:36:23.750 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:6cac518f:::0:head 2023-12-07T07:36:23.750 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:6cac518f:::0:head -> 0 2023-12-07T07:36:23.750 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=15 cursor=18:863748b0:::15:head 2023-12-07T07:36:23.750 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:863748b0:::15:head 2023-12-07T07:36:23.750 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:863748b0:::15:head -> 15 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=11 cursor=18:89d3ae78:::11:head 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:89d3ae78:::11:head 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:89d3ae78:::11:head -> 11 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=5 cursor=18:b29083e3:::5:head 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:b29083e3:::5:head 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:b29083e3:::5:head -> 5 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=8 cursor=18:bd63b0f1:::8:head 2023-12-07T07:36:23.751 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:bd63b0f1:::8:head 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:bd63b0f1:::8:head -> 8 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=6 cursor=18:c4fdafeb:::6:head 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:c4fdafeb:::6:head 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:c4fdafeb:::6:head -> 6 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=3 cursor=18:cfc208b3:::3:head 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:cfc208b3:::3:head 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:cfc208b3:::3:head -> 3 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=4 cursor=18:d83876eb:::4:head 2023-12-07T07:36:23.752 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:d83876eb:::4:head 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:d83876eb:::4:head -> 4 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=12 cursor=18:de5d7c5f:::12:head 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:de5d7c5f:::12:head 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:de5d7c5f:::12:head -> 12 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=9 cursor=18:e960b815:::9:head 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:e960b815:::9:head 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:e960b815:::9:head -> 9 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : oid=2 cursor=18:f905c69b:::2:head 2023-12-07T07:36:23.753 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:f905c69b:::2:head 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:f905c69b:::2:head -> 2 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:52ea6a34:::10:head 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:52ea6a34:::10:head expected=18:52ea6a34:::10:head 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:52ea6a34:::10:head -> 10 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=10 expected=10 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:f905c69b:::2:head 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:f905c69b:::2:head expected=18:f905c69b:::2:head 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:f905c69b:::2:head -> 2 2023-12-07T07:36:23.754 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=2 expected=2 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:e960b815:::9:head 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:e960b815:::9:head expected=18:e960b815:::9:head 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:e960b815:::9:head -> 9 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=9 expected=9 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:de5d7c5f:::12:head 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:de5d7c5f:::12:head expected=18:de5d7c5f:::12:head 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:de5d7c5f:::12:head -> 12 2023-12-07T07:36:23.755 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=12 expected=12 2023-12-07T07:36:23.756 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:d83876eb:::4:head 2023-12-07T07:36:23.756 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:d83876eb:::4:head expected=18:d83876eb:::4:head 2023-12-07T07:36:23.756 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:d83876eb:::4:head -> 4 2023-12-07T07:36:23.756 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=4 expected=4 2023-12-07T07:36:23.756 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:cfc208b3:::3:head 2023-12-07T07:36:23.756 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:cfc208b3:::3:head expected=18:cfc208b3:::3:head 2023-12-07T07:36:23.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:cfc208b3:::3:head -> 3 2023-12-07T07:36:23.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=3 expected=3 2023-12-07T07:36:23.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:c4fdafeb:::6:head 2023-12-07T07:36:23.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:c4fdafeb:::6:head expected=18:c4fdafeb:::6:head 2023-12-07T07:36:23.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:c4fdafeb:::6:head -> 6 2023-12-07T07:36:23.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=6 expected=6 2023-12-07T07:36:23.757 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:bd63b0f1:::8:head 2023-12-07T07:36:23.758 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:bd63b0f1:::8:head expected=18:bd63b0f1:::8:head 2023-12-07T07:36:23.758 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:bd63b0f1:::8:head -> 8 2023-12-07T07:36:24.195 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:50.643 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:24.201 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:50.651 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:24.388 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:03:50.826 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:24.404 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:03:50.842 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:24.630 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:51.079 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:24.680 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:03:51.118 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:36:31.696 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : api_stat: Running main() from gmock_main.cc 2023-12-07T07:36:31.696 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [----------] Global test environment set-up. 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStat.Stat (15533 ms) 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6710 ms) 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStat.StatNS (26420 ms) 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (154 ms) 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (985 ms) 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [----------] 5 tests from LibRadosStat (49802 ms total) 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: 2023-12-07T07:36:31.697 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (10260 ms) 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (12475 ms) 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (3072 ms) 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1285 ms) 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (27092 ms total) 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [----------] Global test environment tear-down 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (111401 ms total) 2023-12-07T07:36:31.698 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat: [ PASSED ] 9 tests. 2023-12-07T07:36:33.586 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: Running main() from gmock_main.cc 2023-12-07T07:36:33.586 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-12-07T07:36:33.586 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [----------] Global test environment set-up. 2023-12-07T07:36:33.586 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-12-07T07:36:33.586 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-12-07T07:36:33.586 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (664 ms) 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (74 ms) 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (92596 ms) 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:09.167235+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi097-92795-4"}]': finished 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:10.517950+0000 mon.c [INF] from='client.? 172.21.15.97:0/699886715' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi097-92795-4", "tierpool": "test-rados-api-smithi097-92795-4-cache"}]: dispatch 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:12.065308+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi097-92795-4", "tierpool": "test-rados-api-smithi097-92795-4-cache"}]: dispatch 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:12.209571+0000 mon.a [INF] from='client.? 172.21.15.97:0/3860548513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi097-92756-10"}]: dispatch 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:12.816832+0000 mon.a [INF] from='client.? 172.21.15.97:0/3860548513' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi097-92756-10"}]: dispatch 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:13.308646+0000 mon.a [WRN] Health check update: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:13.375395+0000 mon.a [INF] from='client.? 172.21.15.97:0/3860548513' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi097-92756-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:14.386027+0000 mon.c [INF] from='client.4664 172.21.15.97:0/2758257523' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi097-92898-7"}]: dispatch 2023-12-07T07:36:33.587 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:15.080448+0000 mon.a [INF] from='client.4664 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi097-92898-7"}]: dispatch 2023-12-07T07:36:33.588 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:15.578236+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi097-92795-4", "tierpool": "test-rados-api-smithi097-92795-4-cache"}]': finished 2023-12-07T07:36:33.588 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:15.734117+0000 mon.c [INF] from='client.? 172.21.15.97:0/3154532325' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-07T07:36:33.588 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:16.599149+0000 mon.a [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-07T07:36:33.588 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:17.506600+0000 client.admin [INF] onexx 2023-12-07T07:36:33.588 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:17.522556+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-07T07:36:33.588 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:21.111293+0000 mon.c [INF] from='client.? 172.21.15.97:0/2568760168' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi097-92627-22"}]: dispatch 2023-12-07T07:36:34.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:21.259767+0000 mon.c [INF] from='client.4664 172.21.15.97:0/2758257523' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi097-92898-7"}]: dispatch 2023-12-07T07:36:34.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:21.741427+0000 mon.c [INF] from='client.? 172.21.15.97:0/3542367096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi097-92587-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T07:36:34.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:22.779593+0000 mon.b [INF] from='client.? 172.21.15.97:0/3514185508' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi097-93059-4"}]: dispatch 2023-12-07T07:36:34.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:22.931435+0000 mon.c [INF] from='client.? 172.21.15.97:0/3154532325' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-07T07:36:34.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:23.062627+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi097-93273-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-07T07:36:34.183 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:23.064627+0000 mon.a [INF] from='client.4676 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi097-92950-7"}]': finished 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:23.066517+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi097-92735-10"}]': finished 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:23.553410+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi097-92627-22"}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:23.578020+0000 mon.a [INF] from='client.4664 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi097-92898-7"}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:23.600882+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi097-92587-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:24.198078+0000 mon.a [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:25.257961+0000 client.admin [INF] twoxx 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:25.298176+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:25.447424+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi097-93059-4"}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:25.628145+0000 mon.b [INF] from='client.? 172.21.15.97:0/584783121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi097-92609-16"}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:25.788697+0000 mon.c [INF] from='client.4676 172.21.15.97:0/517968738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi097-92950-7"}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:25.792799+0000 mon.c [INF] from='client.? 172.21.15.97:0/3096547870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi097-92735-10"}]: dispatch 2023-12-07T07:36:34.184 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:25.806069+0000 mon.c [INF] from='client.? 172.21.15.97:0/2568760168' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi097-92627-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:26.105441+0000 mon.b [INF] from='client api_lock_pp: Running main() from gmock_main.cc 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: seed 92735 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (20178 ms) 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (5847 ms) 2023-12-07T07:36:34.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (2547 ms) 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (2146 ms) 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (2854 ms) 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (26446 ms) 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5282 ms) 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (1853 ms) 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (67153 ms total) 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-12-07T07:36:34.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (8909 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (616 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1270 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1146 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (1845 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (364 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (246 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (158 ms) 2023-12-07T07:36:34.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (14554 ms total) 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (114363 ms total) 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-12-07T07:36:34.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: seed 92950 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (30775 ms) 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (18184 ms) 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (18 ms) 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (509 ms) 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (15803 ms) 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (65290 ms total) 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-12-07T07:36:34.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (9235 ms) 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1292 ms) 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (3675 ms) 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (1559 ms) 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (15761 ms total) 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (114280 ms total) 2023-12-07T07:36:34.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-12-07T07:36:37.739 INFO:tasks.workunit.client.0.smithi097.stdout:.? 172.21.15.97:0/3305574071' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi097-92574-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T07:36:37.739 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:26.527834+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi097-92609-16"}]: dispatch 2023-12-07T07:36:37.739 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:26.552760+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi097-92574-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:26.943669+0000 mon.b [INF] from='client.? 172.21.15.97:0/1143427966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi097-93048-12"}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:27.256240+0000 mon.a [INF] from='client.4664 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi097-92898-7"}]': finished 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:27.258792+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi097-92587-8","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:27.589401+0000 mon.a [INF] from='client.4676 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi097-92950-7"}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:27.657951+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi097-92735-10"}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:27.666840+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi097-92627-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:29.545686+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:29.780766+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi097-93048-12"}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:29.291312+0000 mon.b [INF] from='client.? 172.21.15.97:0/584783121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi097-92609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:30.688267+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi097-92609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:31.257433+0000 mon.a [INF] from='client.? 172.21.15.97:0/3860548513' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi097-92756-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi097-92756-10"}]': finished 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:31.276887+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi097-93059-4"}]': finished 2023-12-07T07:36:37.740 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:31.278694+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi097-92574-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:31.281031+0000 mon.a [INF] from='client.4676 ' entity='client.admin' cmd='[{ list_parallel: process_1_[93326]: starting. 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_1_[93326]: creating pool ceph_test_rados_list_parallel.smithi097-93298 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_1_[93326]: created object 0... 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_1_[93326]: created object 25... 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_1_[93326]: created object 49... 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_1_[93326]: finishing. 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_1_[93326]: shutting down. 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_2_[93328]: starting. 2023-12-07T07:36:41.373 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_2_[93328]: listing objects. 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_2_[93328]: listed object 0... 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_2_[93328]: listed object 25... 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_2_[93328]: saw 50 objects 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_2_[93328]: shutting down. 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_3_[107861]: starting. 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_3_[107861]: creating pool ceph_test_rados_list_parallel.smithi097-93298 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_3_[107861]: created object 0... 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_3_[107861]: created object 25... 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_3_[107861]: created object 49... 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_3_[107861]: finishing. 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_3_[107861]: shutting down. 2023-12-07T07:36:41.374 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_4_[107862]: starting. 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_4_[107862]: listing objects. 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_4_[107862]: listed object 0... 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_4_[107862]: listed object 25... 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_4_[107862]: saw 38 objects 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_4_[107862]: shutting down. 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_5_[107863]: starting. 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_5_[107863]: removed 25 objects... 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_5_[107863]: removed half of the objects 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_5_[107863]: removed 50 objects... 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_5_[107863]: removed 50 objects 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_5_[107863]: shutting down. 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.375 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_6_[118485]: starting. 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_6_[118485]: creating pool ceph_test_rados_list_parallel.smithi097-93298 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_6_[118485]: created object 0... 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_6_[118485]: created object 25... 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_6_[118485]: created object 49... 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_6_[118485]: finishing. 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_6_[118485]: shutting down. 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_7_[118486]: starting. 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_7_[118486]: listing objects. 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_7_[118486]: listed object 0... 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_7_[118486]: listed object 25... 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_7_[118486]: listed object 50... 2023-12-07T07:36:41.376 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_7_[118486]: saw 53 objects 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_7_[118486]: shutting down. 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_8_[118487]: starting. 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_8_[118487]: added 25 objects... 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_8_[118487]: added half of the objects 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_8_[118487]: added 50 objects... 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_8_[118487]: added 50 objects 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_8_[118487]: shutting down. 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:41.377 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.806 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.806 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_9_[127279]: starting. 2023-12-07T07:36:50.806 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_9_[127279]: creating pool ceph_test_rados_list_parallel.smithi097-93298 2023-12-07T07:36:50.806 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_9_[127279]: created object 0... 2023-12-07T07:36:50.806 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_9_[127279]: created object 25... 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_9_[127279]: created object 49... 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_9_[127279]: finishing. 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_9_[127279]: shutting down. 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: starting. 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: listing objects. 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: listed object 0... 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: listed object 25... 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: listed object 50... 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: listed object 75... 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: saw 96 objects 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_10_[127280]: shutting down. 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.807 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_12_[127282]: starting. 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_12_[127282]: added 25 objects... 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_12_[127282]: added half of the objects 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_12_[127282]: added 50 objects... 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_12_[127282]: added 50 objects 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_12_[127282]: shutting down. 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_11_[127281]: starting. 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_11_[127281]: added 25 objects... 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_11_[127281]: added half of the objects 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_11_[127281]: added 50 objects... 2023-12-07T07:36:50.808 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_11_[127281]: added 50 objects 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_11_[127281]: shutting down. 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_13_[127283]: starting. 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_13_[127283]: removed 25 objects... 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_13_[127283]: removed half of the objects 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_13_[127283]: removed 50 objects... 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_13_[127283]: removed 50 objects 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_13_[127283]: shutting down. 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.809 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_14_[138544]: starting. 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_14_[138544]: creating pool ceph_test_rados_list_parallel.smithi097-93298 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_14_[138544]: created object 0... 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_14_[138544]: created object 25... 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_14_[138544]: created object 49... 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_14_[138544]: finishing. 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_14_[138544]: shutting down. 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: starting. 2023-12-07T07:36:50.810 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: listing objects. 2023-12-07T07:36:55.266 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: delete_pools_parallel: process_1_[93324]: starting. 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_1_[93324]: creating pool ceph_test_rados_delete_pools_parallel.smithi097-93311 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_1_[93324]: created object 0... 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_1_[93324]: created object 25... 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_1_[93324]: created object 49... 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_1_[93324]: finishing. 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_1_[93324]: shutting down. 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_2_[93325]: starting. 2023-12-07T07:36:55.267 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_2_[93325]: deleting pool ceph_test_rados_delete_pools_parallel.smithi097-93311 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_2_[93325]: shutting down. 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: ******************************* 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_3_[112580]: starting. 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_3_[112580]: creating pool ceph_test_rados_delete_pools_parallel.smithi097-93311 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_3_[112580]: created object 0... 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_3_[112580]: created object 25... 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_3_[112580]: created object 49... 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_3_[112580]: finishing. 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_3_[112580]: shutting down. 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: ******************************* 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_5_[112582]: starting. 2023-12-07T07:36:55.268 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_5_[112582]: listing objects. 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_5_[112582]: listed object 0... 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_5_[112582]: listed object 25... 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_5_[112582]: saw 50 objects 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_5_[112582]: shutting down. 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: ******************************* 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_4_[112581]: starting. 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_4_[112581]: deleting pool ceph_test_rados_delete_pools_parallel.smithi097-93311 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: process_4_[112581]: shutting down. 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: ******************************* 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: ******************************* 2023-12-07T07:36:55.269 INFO:tasks.workunit.client.0.smithi097.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-12-07T07:36:55.450 INFO:tasks.workunit.client.0.smithi097.stdout:"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi097-92950-7"}]': finished 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:31.282869+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi097-92735-10"}]': finished 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:31.285563+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi097-92627-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:32.281997+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:32.891317+0000 mon.c [INF] from='client.? 172.21.15.97:0/2568760168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi097-92627-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi097-92627-22"}]: dispatch 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:33.052996+0000 mon.b [INF] from='client.? 172.21.15.97:0/1143427966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi097-93048-12"}]: dispatch 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:33.187212+0000 mon.c [INF] from='client.? 172.21.15.97:0/3154532325' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:33.399193+0000 mon.b [INF] from='client.? 172.21.15.97:0/3514185508' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi097-93059-4", "tierpool": "test-rados-api-smithi097-93059-6"}]: dispatch 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:34.387341+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi097-92627-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi097-92627-22"}]: dispatch 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:34.407968+0000 client.admin [INF] threexx 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:34.423550+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:34.668684+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi097-93048-12"}]: dispatch 2023-12-07T07:36:55.451 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:34.701195+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi097-93059-4", "tierpool": "test-rados-api-smithi097-93059-6"}]: dispatch 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:34.744383+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi097-92609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: got: 2023-12-07T07:36:35.152355+0000 mon.a [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (41710 ms) 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (135044 ms total) 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [----------] Global test environment tear-down 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (135044 ms total) 2023-12-07T07:36:55.452 INFO:tasks.workunit.client.0.smithi097.stdout: api_cmd: [ PASSED ] 4 tests. 2023-12-07T07:36:58.002 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: Running main() from gmock_main.cc 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [----------] Global test environment set-up. 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: seed 92627 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-12-07T07:36:58.003 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (17266 ms) 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (7253 ms) 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4992 ms) 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1335 ms) 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3100 ms) 2023-12-07T07:36:58.004 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-12-07T07:36:58.005 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (1583 ms) 2023-12-07T07:36:58.005 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-12-07T07:36:58.005 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2081 ms) 2023-12-07T07:36:58.005 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-12-07T07:36:58.005 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (4638 ms) 2023-12-07T07:36:58.005 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-12-07T07:36:58.006 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (2891 ms) 2023-12-07T07:36:58.006 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-12-07T07:36:58.006 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (6430 ms) 2023-12-07T07:36:58.006 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-12-07T07:36:58.006 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8315 ms) 2023-12-07T07:36:58.007 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-12-07T07:36:58.007 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (2939 ms) 2023-12-07T07:36:58.007 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-12-07T07:36:58.007 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (5737 ms) 2023-12-07T07:36:58.007 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-12-07T07:36:58.007 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (6504 ms) 2023-12-07T07:36:58.007 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (4531 ms) 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (1124 ms) 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (111 ms) 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (183 ms) 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-12-07T07:36:58.008 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (202 ms) 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (81215 ms total) 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (6240 ms) 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (846 ms) 2023-12-07T07:36:58.009 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-12-07T07:36:58.010 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (248 ms) 2023-12-07T07:36:58.010 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-12-07T07:36:58.010 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (237 ms) 2023-12-07T07:36:58.010 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.R open_pools_parallel: process_1_[93354]: starting. 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_1_[93354]: creating pool ceph_test_rados_open_pools_parallel.smithi097-93314 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_1_[93354]: created object 0... 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_1_[93354]: created object 25... 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_1_[93354]: created object 49... 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_1_[93354]: finishing. 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_1_[93354]: shutting down. 2023-12-07T07:36:58.918 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_2_[93358]: starting. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_2_[93358]: rados_pool_create. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_2_[93358]: rados_ioctx_create. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_2_[93358]: shutting down. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: ******************************* 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_3_[111713]: starting. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_3_[111713]: creating pool ceph_test_rados_open_pools_parallel.smithi097-93314 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_3_[111713]: created object 0... 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_3_[111713]: created object 25... 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_3_[111713]: created object 49... 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_3_[111713]: finishing. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_3_[111713]: shutting down. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: ******************************* 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_4_[111714]: starting. 2023-12-07T07:36:58.919 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_4_[111714]: rados_pool_create. 2023-12-07T07:36:58.920 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_4_[111714]: rados_ioctx_create. 2023-12-07T07:36:58.920 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: process_4_[111714]: shutting down. 2023-12-07T07:36:58.920 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: ******************************* 2023-12-07T07:36:58.920 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: ******************************* 2023-12-07T07:36:58.920 INFO:tasks.workunit.client.0.smithi097.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-12-07T07:37:02.849 INFO:tasks.workunit.client.0.smithi097.stdout:ushing 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: flushed 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (4679 ms) 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: trying... 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227122469728 notify_id 412316860422 notifier_gid 4720 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: timed out 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: flushing 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: flushed 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (4452 ms) 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: List watches 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: notify2 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227122470768 notify_id 416611827719 notifier_gid 4720 2023-12-07T07:37:02.850 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: notify2 done 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: watch_check 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: unwatch2 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: flushing 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: done 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (4326 ms) 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: List watches 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: notify2 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: handle_notify cookie 94227122470768 notify_id 420906795016 notifier_gid 4720 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: notify2 done 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: watch_check 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: unwatch2 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: flushing 2023-12-07T07:37:02.851 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: done 2023-12-07T07:37:02.852 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3239 ms) 2023-12-07T07:37:02.852 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (60179 ms total) 2023-12-07T07:37:02.852 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: 2023-12-07T07:37:02.852 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-12-07T07:37:02.852 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (142485 ms total) 2023-12-07T07:37:02.852 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-12-07T07:37:04.554 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: Running main() from gmock_main.cc 2023-12-07T07:37:04.554 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-12-07T07:37:04.554 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [----------] Global test environment set-up. 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (26306 ms) 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.LockShared (14587 ms) 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (3823 ms) 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-12-07T07:37:04.555 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (7334 ms) 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (10433 ms) 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5106 ms) 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (10373 ms) 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (658 ms) 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [----------] 8 tests from LibRadosLock (78620 ms total) 2023-12-07T07:37:04.556 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (7825 ms) 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (608 ms) 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1161 ms) 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1618 ms) 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (622 ms) 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-12-07T07:37:04.557 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (519 ms) 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (359 ms) 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (132 ms) 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (12844 ms total) 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [----------] Global test environment tear-down 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (144320 ms total) 2023-12-07T07:37:04.558 INFO:tasks.workunit.client.0.smithi097.stdout: api_lock: [ PASSED ] 16 tests. 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: listed object 0... 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: listed object 25... 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: listed object 50... 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: listed object 75... 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: listed object 100... 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: listed object 125... 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: saw 150 objects 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_15_[138545]: shutting down. 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.956 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_16_[138546]: starting. 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_16_[138546]: added 25 objects... 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_16_[138546]: added half of the objects 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_16_[138546]: added 50 objects... 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_16_[138546]: added 50 objects 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: process_16_[138546]: shutting down. 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******************************* 2023-12-07T07:37:04.957 INFO:tasks.workunit.client.0.smithi097.stdout: list_parallel: ******* SUCCESS ********** 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.Exec (17647 ms) 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (22290 ms) 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (1182 ms) 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4431 ms) 2023-12-07T07:37:07.228 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (68903 ms) 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] 8 tests from LibRadosMisc (114453 ms total) 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [----------] Global test environment tear-down 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (146964 ms total) 2023-12-07T07:37:07.229 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc: [ PASSED ] 12 tests. 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout:oundTripPP2 (140 ms) 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (300 ms) 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (223 ms) 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (220 ms) 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-12-07T07:37:08.310 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (215 ms) 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (155 ms) 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (189 ms) 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (162 ms) 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (486 ms) 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (194 ms) 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (289 ms) 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-12-07T07:37:08.311 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (124 ms) 2023-12-07T07:37:08.312 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-12-07T07:37:08.312 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (142 ms) 2023-12-07T07:37:08.312 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (10410 ms total) 2023-12-07T07:37:08.312 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: 2023-12-07T07:37:08.312 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [----------] Global test environment tear-down 2023-12-07T07:37:08.312 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (148132 ms total) 2023-12-07T07:37:08.312 INFO:tasks.workunit.client.0.smithi097.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-12-07T07:37:10.875 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:04:37.313 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:10.876 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:04:37.314 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:10.883 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:37.332 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:10.886 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:37.335 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:10.956 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:04:37.394 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:10.956 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:37.405 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:10.959 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:37.409 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:10.997 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:37.446 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:11.065 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:37.514 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify_test_cb 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (62728 ms) 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-12-07T07:37:14.127 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94013097806336 err -107 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (15223 ms) 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94013097806336 err -107 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (2696 ms) 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925824 cookie 94013099486112 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (228 ms) 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925824 cookie 94013099486112 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (399 ms) 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925826 cookie 94013100211600 2023-12-07T07:37:14.128 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (297 ms) 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925824 cookie 94013100211600 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925824 cookie 94013100236672 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (380 ms) 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925826 cookie 94013100236672 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925826 cookie 94013100236672 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 403726925827 cookie 94013100236672 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 408021893125 cookie 94013100236672 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3171 ms) 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94013100236672 err -107 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_cb from 4440 notify_id 416611827715 cookie 94013100240144 2023-12-07T07:37:14.129 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (7600 ms) 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94013100240144 err -107 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (495 ms) 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (93217 ms total) 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: watch_notify_test_cb 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (6449 ms) 2023-12-07T07:37:14.130 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (6449 ms total) 2023-12-07T07:37:14.131 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: 2023-12-07T07:37:14.131 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-12-07T07:37:14.131 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (153759 ms total) 2023-12-07T07:37:14.131 INFO:tasks.workunit.client.0.smithi097.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: Running main() from gmock_main.cc 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [----------] Global test environment set-up. 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (29396 ms) 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (11948 ms) 2023-12-07T07:37:16.991 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (4708 ms) 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.Checksum (990 ms) 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (1477 ms) 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (1929 ms) 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (2928 ms) 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1347 ms) 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.TruncTest (3405 ms) 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-12-07T07:37:16.992 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (2933 ms) 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3990 ms) 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.RmXattr (12152 ms) 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIo.XattrIter (3320 ms) 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [----------] 14 tests from LibRadosIo (80523 ms total) 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (9382 ms) 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (577 ms) 2023-12-07T07:37:16.993 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (168 ms) 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (205 ms) 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (2118 ms) 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (2948 ms) 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (819 ms) 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-12-07T07:37:16.994 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (2189 ms) 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (1118 ms) 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (243 ms) 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [----------] 10 tests from LibRadosIoEC (19767 ms total) 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [----------] Global test environment tear-down 2023-12-07T07:37:16.995 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [==========] 24 tests from 2 test suites ran. (156820 ms total) 2023-12-07T07:37:16.996 INFO:tasks.workunit.client.0.smithi097.stdout: api_io: [ PASSED ] 24 tests. 2023-12-07T07:37:51.124 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:17.572 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.126 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:17.575 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.161 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:17.610 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.186 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:17.635 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.190 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:17.628 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.192 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:17.630 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.259 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:17.708 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.305 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:17.754 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:51.308 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:17.746 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:37:55.297 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: Running main() from gmock_main.cc 2023-12-07T07:37:55.297 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-12-07T07:37:55.297 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] Global test environment set-up. 2023-12-07T07:37:55.297 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-12-07T07:37:55.297 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-12-07T07:37:55.297 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (31225 ms) 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4747 ms) 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (24750 ms) 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (9136 ms) 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (69858 ms total) 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (32461 ms) 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (39921 ms) 2023-12-07T07:37:55.298 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (7202 ms) 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (79584 ms total) 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (4126 ms) 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2700 ms) 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2905 ms) 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2221 ms) 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (11952 ms total) 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: 2023-12-07T07:37:55.299 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (7713 ms) 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4805 ms) 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (12519 ms total) 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [----------] Global test environment tear-down 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (194950 ms total) 2023-12-07T07:37:55.300 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-12-07T07:38:04.282 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-12-07T07:38:04.282 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-12-07T07:38:04.282 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-12-07T07:38:04.282 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-12-07T07:38:04.282 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: seed 92870 2023-12-07T07:38:04.282 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (29339 ms) 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3807 ms) 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (14733 ms) 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (3115 ms) 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (15312 ms) 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (66306 ms total) 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-12-07T07:38:04.283 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (36887 ms) 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (36257 ms) 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (9588 ms) 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4305 ms) 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3104 ms) 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-64-ge068ebcf/rpm/el9/BUILD/ceph-19.0.0-64-ge068ebcf/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-12-07T07:38:04.284 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi097-92870-7 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: waiting for snaps to purge 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18903 ms) 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (109044 ms total) 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (4025 ms) 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2681 ms) 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3346 ms) 2023-12-07T07:38:04.285 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-12-07T07:38:04.286 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2329 ms) 2023-12-07T07:38:04.286 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (12381 ms total) 2023-12-07T07:38:04.286 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: 2023-12-07T07:38:04.286 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-12-07T07:38:11.347 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:37.796 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.351 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:37.800 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.414 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:37.864 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.415 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:37.864 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.416 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:37.865 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.421 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:37.859 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.427 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:37.865 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.476 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:37.925 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:11.503 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:37.941 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:18.899 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] L api_aio: Running main() from gmock_main.cc 2023-12-07T07:38:18.900 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-12-07T07:38:18.900 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [----------] Global test environment set-up. 2023-12-07T07:38:18.900 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-12-07T07:38:18.900 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-12-07T07:38:18.900 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4500 ms) 2023-12-07T07:38:18.900 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-12-07T07:38:18.900 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (30830 ms) 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (21114 ms) 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (6274 ms) 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (11292 ms) 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (20096 ms) 2023-12-07T07:38:18.901 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (30383 ms) 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (22562 ms) 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4620 ms) 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (6117 ms) 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (4257 ms) 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4717 ms) 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-12-07T07:38:18.902 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3250 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2900 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.Flush (2879 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4301 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4802 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4380 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3759 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3327 ms) 2023-12-07T07:38:18.903 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3410 ms) 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4394 ms) 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4191 ms) 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3752 ms) 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3363 ms) 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3267 ms) 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [----------] 26 tests from LibRadosAio (218737 ms total) 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: 2023-12-07T07:38:18.904 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleibRadosSnapshotsSelfManagedECPP.SnapPP 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4935 ms) 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (6403 ms) 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4369 ms) 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (15707 ms total) 2023-12-07T07:38:23.437 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: 2023-12-07T07:38:23.438 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-12-07T07:38:23.438 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (223153 ms total) 2023-12-07T07:38:23.438 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-12-07T07:38:23.438 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-12-07T07:38:23.438 INFO:tasks.workunit.client.0.smithi097.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: Running main() from gmock_main.cc 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4460 ms) 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (44494 ms) 2023-12-07T07:38:27.772 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (15427 ms) 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (7475 ms) 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (19240 ms) 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (28788 ms) 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (26216 ms) 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (4256 ms) 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3606 ms) 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-12-07T07:38:27.773 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3824 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3169 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3118 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (4725 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3073 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3900 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (4310 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (4826 ms) 2023-12-07T07:38:27.774 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4352 ms) 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3768 ms) 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3330 ms) 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (5677 ms) 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3249 ms) 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2980 ms) 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (6094 ms) 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-12-07T07:38:27.775 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (5462 ms) 2023-12-07T07:38:27.776 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-12-07T07:38:27.776 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (4518 ms) 2023-12-07T07:38:27.776 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-12-07T07:38:31.313 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:57.762 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.377 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:57.826 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.389 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:57.838 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.394 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:57.844 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.433 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:57.870 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.438 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:57.876 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.466 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:57.915 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.476 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:57.925 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:31.520 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:05:57.957 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.340 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:17.789 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.345 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:17.794 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.401 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:17.850 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.406 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:17.855 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.419 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:17.868 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.422 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:17.860 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.429 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:17.867 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.474 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:17.923 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:38:51.486 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:17.924 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.594 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:38.042 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.601 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:38.050 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.610 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:38.048 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.611 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:38.050 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.612 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:38.061 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.616 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:38.065 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.756 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:38.205 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.791 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:38.229 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:11.863 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:38.312 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:17.137 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadoentry=8 expected=8 2023-12-07T07:39:17.137 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:b29083e3:::5:head 2023-12-07T07:39:17.137 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:b29083e3:::5:head expected=18:b29083e3:::5:head 2023-12-07T07:39:17.137 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:b29083e3:::5:head -> 5 2023-12-07T07:39:17.137 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=5 expected=5 2023-12-07T07:39:17.137 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:566253c9:::13:head 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:566253c9:::13:head expected=18:566253c9:::13:head 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:566253c9:::13:head -> 13 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=13 expected=13 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:89d3ae78:::11:head 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:89d3ae78:::11:head expected=18:89d3ae78:::11:head 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:89d3ae78:::11:head -> 11 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=11 expected=11 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:863748b0:::15:head 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:863748b0:::15:head expected=18:863748b0:::15:head 2023-12-07T07:39:17.138 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:863748b0:::15:head -> 15 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=15 expected=15 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:6cac518f:::0:head 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:6cac518f:::0:head expected=18:6cac518f:::0:head 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:6cac518f:::0:head -> 0 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=0 expected=0 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:62a1935d:::14:head 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:62a1935d:::14:head expected=18:62a1935d:::14:head 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:62a1935d:::14:head -> 14 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=14 expected=14 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:5c6b0b28:::7:head 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:5c6b0b28:::7:head expected=18:5c6b0b28:::7:head 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:5c6b0b28:::7:head -> 7 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=7 expected=7 2023-12-07T07:39:17.139 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : seek to 18:02547ec2:::1:head 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : cursor()=18:02547ec2:::1:head expected=18:02547ec2:::1:head 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: > 18:02547ec2:::1:head -> 1 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: : entry=1 expected=1 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (9026 ms) 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (99082 ms) 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: split 0/5 -> MIN 18:33333333::::head 2023-12-07T07:39:17.140 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: split 1/5 -> 18:33333333::::head 18:66666666::::head 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: split 2/5 -> 18:66666666::::head 18:99999999::::head 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: split 3/5 -> 18:99999999::::head 18:cccccccc::::head 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: split 4/5 -> 18:cccccccc::::head MAX 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (62127 ms) 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] 7 tests from LibRadosList (263624 ms total) 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1967 ms) 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset foo1,foo2,foo3 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo1 2023-12-07T07:39:17.141 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo2 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo3 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset foo1,foo4,foo5 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo4 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo5 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo1 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset foo6,foo7 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo7 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: foo6 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-07T07:39:17.142 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns1:foo4 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns1:foo5 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns2:foo7 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns2:foo6 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: ns1:foo1 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: :foo1 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: :foo2 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: :foo3 2023-12-07T07:39:17.143 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3392 ms) 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRad api_c_write_operations: Running main() from gmock_main.cc 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-12-07T07:39:21.999 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (25499 ms) 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (9540 ms) 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (11348 ms) 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (25097 ms) 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (14461 ms) 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (30634 ms) 2023-12-07T07:39:22.000 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-12-07T07:39:22.001 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (164974 ms) 2023-12-07T07:39:22.001 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (281553 ms total) 2023-12-07T07:39:22.001 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: 2023-12-07T07:39:22.001 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-12-07T07:39:22.001 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (281553 ms total) 2023-12-07T07:39:22.001 INFO:tasks.workunit.client.0.smithi097.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout:osListEC.ListObjectsStart 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 1 0 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 10 0 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 13 0 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 7 0 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 14 0 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 0 0 2023-12-07T07:39:28.822 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 15 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 11 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 5 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 8 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 6 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 3 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 4 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 12 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 9 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 2 0 2023-12-07T07:39:28.823 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (5764 ms) 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] 3 tests from LibRadosListEC (11123 ms total) 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3704 ms) 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] 1 test from LibRadosListNP (3704 ms total) 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: 2023-12-07T07:39:28.824 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [----------] Global test environment tear-down 2023-12-07T07:39:28.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [==========] 11 tests from 3 test suites ran. (288589 ms total) 2023-12-07T07:39:28.825 INFO:tasks.workunit.client.0.smithi097.stdout: api_list: [ PASSED ] 11 tests. 2023-12-07T07:39:31.606 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:58.055 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.608 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:58.058 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.645 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:58.083 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.648 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:58.086 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.656 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:58.105 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.658 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:58.107 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.670 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:58.119 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.693 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:06:58.130 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:31.707 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:58.156 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:35.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-12-07T07:39:35.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-12-07T07:39:35.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-12-07T07:39:35.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-12-07T07:39:35.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: seed 92939 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (270 ms) 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (26506 ms) 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (5768 ms) 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (12113 ms) 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (6788 ms) 2023-12-07T07:39:35.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4374 ms) 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (4834 ms) 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (1858 ms) 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1644 ms) 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (3859 ms) 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (1484 ms) 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (3921 ms) 2023-12-07T07:39:35.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (5366 ms) 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: osd_max_attr_size = 0 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (114372 ms) 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (14131 ms) 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: done waiting, doing copies 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: waiting for final deep scrubs... 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: done waiting 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (78291 ms) 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-12-07T07:39:35.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (102 ms) 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (347 ms) 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5061 ms) 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-12-07T07:39:35.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-12-07T07:39:46.567 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:13.016 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.647 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:13.096 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.690 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:13.128 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.697 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:13.146 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.699 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:13.137 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.706 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:13.155 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.717 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:13.166 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.749 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:13.187 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:46.754 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:13.204 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.766 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:18.215 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.768 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:18.217 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.814 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:18.252 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.815 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:18.253 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.826 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:18.275 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.831 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:18.269 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.835 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:18.285 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.837 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:18.286 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:39:51.854 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:18.303 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:04.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (291089 ms total) 2023-12-07T07:40:04.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: 2023-12-07T07:40:04.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-12-07T07:40:04.245 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-12-07T07:40:04.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2514 ms) 2023-12-07T07:40:04.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2514 ms total) 2023-12-07T07:40:04.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: 2023-12-07T07:40:04.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-12-07T07:40:04.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-12-07T07:40:04.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (477 ms) 2023-12-07T07:40:04.246 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-12-07T07:40:04.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (71 ms) 2023-12-07T07:40:04.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (548 ms total) 2023-12-07T07:40:04.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: 2023-12-07T07:40:04.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-12-07T07:40:04.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-12-07T07:40:04.247 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (508 ms) 2023-12-07T07:40:04.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-12-07T07:40:04.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (76 ms) 2023-12-07T07:40:04.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (584 ms total) 2023-12-07T07:40:04.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: 2023-12-07T07:40:04.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-12-07T07:40:04.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-12-07T07:40:04.248 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (459 ms) 2023-12-07T07:40:04.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-12-07T07:40:04.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (41 ms) 2023-12-07T07:40:04.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (500 ms total) 2023-12-07T07:40:04.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: 2023-12-07T07:40:04.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-12-07T07:40:04.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-12-07T07:40:04.249 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1441 ms) 2023-12-07T07:40:04.250 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1441 ms total) 2023-12-07T07:40:04.250 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: 2023-12-07T07:40:04.250 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-12-07T07:40:04.250 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (323909 ms total) 2023-12-07T07:40:04.250 INFO:tasks.workunit.client.0.smithi097.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-12-07T07:40:11.974 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:38.423 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:11.981 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:38.430 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:12.034 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:38.483 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:12.039 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:38.488 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:12.040 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:38.478 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:12.041 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:38.490 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:12.046 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:38.484 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:12.090 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:38.539 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:12.109 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:38.547 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:27.482 INFO:tasks.workunit.client.0.smithi097.stdout:Write 2023-12-07T07:40:27.482 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7917 ms) 2023-12-07T07:40:27.482 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-12-07T07:40:27.482 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7687 ms) 2023-12-07T07:40:27.482 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-12-07T07:40:27.482 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (11873 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7808 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (8262 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8958 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (9153 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7135 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7567 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6796 ms) 2023-12-07T07:40:27.483 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (8152 ms) 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7728 ms) 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7146 ms) 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6810 ms) 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7572 ms) 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8017 ms) 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (128581 ms total) 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: 2023-12-07T07:40:27.484 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [----------] Global test environment tear-down 2023-12-07T07:40:27.485 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (347318 ms total) 2023-12-07T07:40:27.485 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio: [ PASSED ] 42 tests. 2023-12-07T07:40:32.119 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:58.568 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.127 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:58.576 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.139 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:58.589 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.186 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:58.634 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.188 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:58.626 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.192 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:58.630 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.199 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:58.648 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.239 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:07:58.677 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:32.260 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:58.709 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.197 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:18.646 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.199 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:18.648 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.243 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:18.692 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.247 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:18.685 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.248 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:18.697 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.251 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:18.689 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.253 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:18.702 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.290 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:18.728 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:40:52.298 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:18.747 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:06.875 INFO:tasks.workunit.client.0.smithi097.stdout:sAio.RoundTripAppendPP (3271 ms) 2023-12-07T07:41:06.875 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-12-07T07:41:06.875 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3111 ms) 2023-12-07T07:41:06.875 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-12-07T07:41:06.876 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2571 ms) 2023-12-07T07:41:06.876 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-12-07T07:41:06.876 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3085 ms) 2023-12-07T07:41:06.876 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-12-07T07:41:06.876 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: setting pool EIO 2023-12-07T07:41:06.876 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: max_success 59, min_failed 60 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (8130 ms) 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3940 ms) 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (248446 ms total) 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-12-07T07:41:06.877 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3407 ms) 2023-12-07T07:41:06.878 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-12-07T07:41:06.878 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10414 ms) 2023-12-07T07:41:06.878 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-12-07T07:41:06.878 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (18132 ms) 2023-12-07T07:41:06.878 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-12-07T07:41:06.878 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3503 ms) 2023-12-07T07:41:06.878 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (35456 ms total) 2023-12-07T07:41:06.879 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: 2023-12-07T07:41:06.879 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-12-07T07:41:06.879 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-07T07:41:06.879 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3627 ms) 2023-12-07T07:41:06.879 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3627 ms total) 2023-12-07T07:41:06.879 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: 2023-12-07T07:41:06.879 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14362 ms) 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (8145 ms) 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7748 ms) 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-12-07T07:41:06.880 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7127 ms) 2023-12-07T07:41:06.881 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-12-07T07:41:06.881 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3235 ms) 2023-12-07T07:41:06.881 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-12-07T07:41:06.881 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6934 ms) 2023-12-07T07:41:06.881 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-12-07T07:41:06.881 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6792 ms) 2023-12-07T07:41:06.881 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7351 ms) 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7798 ms) 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7190 ms) 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7060 ms) 2023-12-07T07:41:06.882 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-12-07T07:41:06.883 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7203 ms) 2023-12-07T07:41:06.883 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-12-07T07:41:06.883 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (8237 ms) 2023-12-07T07:41:12.148 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:38.596 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.152 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:38.601 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.198 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:38.636 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.200 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:38.638 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.208 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:38.657 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.214 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:38.663 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.218 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:38.668 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.238 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:38.676 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:12.257 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:38.706 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.237 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:58.686 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.238 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:58.687 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.277 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:58.726 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.285 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:58.723 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.286 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:58.724 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.289 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:58.738 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.290 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:58.740 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.323 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:08:58.761 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:32.328 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:58.778 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.405 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:03.854 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.407 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:03.856 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.422 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:03.860 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.472 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:03.909 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.478 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:03.924 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.486 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:03.935 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.506 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:03.944 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.509 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:03.958 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:37.527 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:03.976 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.463 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:18.912 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.464 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:18.913 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.497 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:18.946 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.513 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:18.951 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.515 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:18.953 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.519 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:18.968 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.521 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:18.970 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.546 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:18.984 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:52.555 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:19.005 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:41:54.569 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3324 ms) 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7015 ms) 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7746 ms) 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6599 ms) 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (8049 ms) 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7105 ms) 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7853 ms) 2023-12-07T07:41:54.570 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (146874 ms total) 2023-12-07T07:41:54.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: 2023-12-07T07:41:54.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-12-07T07:41:54.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (434403 ms total) 2023-12-07T07:41:54.571 INFO:tasks.workunit.client.0.smithi097.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-12-07T07:42:17.377 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:43.825 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.381 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:43.830 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.403 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:43.852 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.434 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:43.871 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.437 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:43.875 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.446 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:43.895 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.450 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:43.899 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.459 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:43.897 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:17.478 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:43.928 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.517 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:03.966 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.576 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:04.025 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.619 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:04.068 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.638 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:04.075 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.638 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:04.087 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.639 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:04.077 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.650 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:04.100 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.680 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:04.118 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:37.692 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:04.141 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.521 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:23.969 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.522 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:23.971 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.556 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:24.005 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.570 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:24.020 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.573 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:24.023 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.574 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:24.012 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.576 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:24.014 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.617 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:24.055 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:42:57.619 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:24.069 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.600 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:44.049 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.602 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:44.051 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.635 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:44.084 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.644 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:44.082 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.646 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:44.084 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.654 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:44.104 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.657 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:44.107 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.684 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:10:44.122 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:17.709 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:44.158 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:25.778 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: seed 92795 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (48270 ms) 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (64414 ms) 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (33 ms) 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (112717 ms total) 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: 2023-12-07T07:43:25.779 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (16272 ms) 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (15824 ms) 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (19669 ms) 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: my_snaps [4] 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: my_snaps [5,4] 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: my_snaps [6,5,4] 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: promoting some heads 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: promoting from clones for snap 7 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: promoting from clones for snap 6 2023-12-07T07:43:25.780 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: promoting from clones for snap 5 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: promoting from clones for snap 4 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: waiting for scrubs... 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: done waiting 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (155458 ms) 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10043 ms) 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9914 ms) 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8832 ms) 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10781 ms) 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-12-07T07:43:25.781 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12104 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10382 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10362 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (39737 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10433 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10230 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16238 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9556 ms) 2023-12-07T07:43:25.782 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-12-07T07:43:25.783 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (14006 ms) 2023-12-07T07:43:37.722 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:04.170 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.725 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:04.174 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.798 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:04.236 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.800 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:04.238 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.832 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:04.281 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.842 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:04.280 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.887 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:04.337 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.891 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:04.341 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:37.942 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:04.391 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.201 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:12.650 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.203 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:12.652 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.236 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:12.685 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.260 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:12.698 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.262 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:12.700 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.282 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:12.731 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.284 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:12.733 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.284 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:12.722 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:46.304 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:12.753 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:47.593 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-12-07T07:43:47.593 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-07T07:43:47.593 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-12-07T07:43:47.593 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9890 ms) 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg_num = 32 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 0 ls 1701935023,0 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 1 ls 1701935023,0 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 2 ls 1701935023,0 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 3 ls 1701935023,0 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 4 ls 1701935023,0 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 5 ls 1701935023,0 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 6 ls 1701935023,0 2023-12-07T07:43:47.594 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 7 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 8 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 9 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 10 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 11 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 12 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 13 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 14 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 15 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 16 ls 1701935023,0 2023-12-07T07:43:47.595 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 17 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 18 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 19 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 20 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 21 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 22 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 23 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 24 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 25 ls 1701935023,0 2023-12-07T07:43:47.596 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 26 ls 1701935023,0 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 27 ls 1701935023,0 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 28 ls 1701935023,0 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 29 ls 1701935023,0 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 30 ls 1701935023,0 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg 31 ls 1701935023,0 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: pg_num = 32 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-12-07T07:43:47.597 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-12-07T07:43:47.598 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-12-07T07:43:47.598 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-12-07T07:43:47.598 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-12-07T07:43:47.598 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-12-07T07:43:47.598 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-12-07T07:43:47.598 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-12-07T07:43:47.598 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-12-07T07:43:47.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-12-07T07:43:47.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-12-07T07:43:47.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-12-07T07:43:47.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-12-07T07:43:47.599 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-12-07T07:43:47.600 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-12-07T07:43:47.601 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-12-07T07:43:47.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-12-07T07:43:47.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-12-07T07:43:47.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-12-07T07:43:47.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-12-07T07:43:47.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-12-07T07:43:47.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-12-07T07:43:47.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-12-07T07:43:47.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-12-07T07:43:47.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-12-07T07:43:47.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-12-07T07:43:47.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-12-07T07:43:47.607 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-12-07T07:43:47.608 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-12-07T07:43:47.609 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-12-07T07:43:47.610 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-12-07T07:43:47.611 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-12-07T07:43:47.612 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-12-07T07:43:47.612 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-12-07T07:43:47.612 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-12-07T07:43:47.612 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-12-07T07:43:47.612 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-12-07T07:43:47.612 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-12-07T07:43:57.665 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:24.114 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.729 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:24.177 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.754 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:24.203 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.785 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:24.223 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.788 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:24.226 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.793 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:24.243 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.798 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:24.247 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.811 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:24.249 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:43:57.824 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:24.273 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.862 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:44.310 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.863 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:44.312 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.895 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:44.344 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.912 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:44.350 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.914 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:44.363 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.915 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:44.365 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.916 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:44.354 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.942 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:11:44.379 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:17.942 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:44.391 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.821 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:04.269 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.823 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:04.273 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.859 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:04.308 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.871 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:04.320 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.874 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:04.312 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.875 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:04.324 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.875 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:04.313 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.908 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:04.346 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:37.913 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:04.362 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.726 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:05.164 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.727 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:05.165 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.740 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:05.178 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.786 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:05.234 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.787 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:05.236 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.802 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:05.251 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.968 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:05.417 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.969 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:05.418 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:38.979 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:05.428 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.869 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:19.318 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.874 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:19.323 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.893 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:19.342 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.920 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:19.370 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.923 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:19.373 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.924 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:19.362 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.926 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:19.364 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.942 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:19.391 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:52.945 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:19.381 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:57.641 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-12-07T07:44:57.642 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-12-07T07:44:57.643 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (20894 ms) 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,0 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: first is 1701935045 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,0 2023-12-07T07:44:57.644 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,1701935050,1701935051,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,1701935050,1701935051,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,1701935050,1701935051,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,1701935050,1701935051,1701935053,1701935054,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,1701935050,1701935051,1701935053,1701935054,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935045,1701935047,1701935048,1701935050,1701935051,1701935053,1701935054,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935048,1701935050,1701935051,1701935053,1701935054,1701935056,1701935057,0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: first now 1701935048, trimmed 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22329 ms) 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: foo0 2023-12-07T07:44:57.645 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-07T07:44:57.646 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (17131 ms) 2023-12-07T07:44:57.646 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-12-07T07:44:58.119 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:24.568 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.122 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:24.572 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.130 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:24.566 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.131 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:24.569 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.147 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:24.584 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.156 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:24.605 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.183 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:24.632 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.190 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:24.640 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:44:58.208 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:24.658 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.889 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:34.338 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.936 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:34.374 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.937 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:34.386 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.939 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:34.377 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.966 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:34.415 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.966 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:34.404 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.987 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:34.436 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:07.990 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:34.439 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:08.014 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:34.463 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:17.968 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:44.417 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:17.969 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:44.419 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:17.985 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:44.434 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:18.012 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:44.461 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:18.015 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:44.453 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:18.018 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:44.467 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:18.018 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:44.456 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:18.033 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:44.482 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:18.033 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:44.471 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.061 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:49.510 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.114 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:49.563 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.131 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:49.580 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.163 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:49.600 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.165 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:49.603 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.172 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:49.621 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.173 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:49.622 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.180 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:49.618 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:23.188 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:49.637 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.103 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:54.552 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.110 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:54.559 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.129 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:54.578 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.163 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:54.612 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.164 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:54.613 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.166 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:54.604 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.169 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:54.607 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.184 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:54.633 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:28.192 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:54.630 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.388 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:04.837 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.391 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:04.840 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.417 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:04.866 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.452 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:04.890 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.456 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:04.905 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.459 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:04.909 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.460 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:04.898 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.493 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:04.943 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:38.504 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:04.942 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.230 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:19.676 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.283 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:19.732 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.311 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:19.760 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.329 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:19.766 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.329 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:19.778 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.332 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:19.769 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.332 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:19.781 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.358 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:19.796 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:53.360 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:19.809 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.280 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:24.728 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.285 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:24.734 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.301 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:24.750 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.329 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:24.767 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.331 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:24.770 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.335 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:24.784 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.337 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:24.787 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.347 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:24.785 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:45:58.360 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:24.809 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.368 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:39.817 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.420 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:39.868 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.438 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:39.886 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.466 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:39.904 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.468 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:39.917 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.469 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:39.907 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.469 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:39.919 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.486 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:39.924 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:13.487 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:39.936 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.346 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:44.795 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.351 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:44.796 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.358 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:44.807 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.388 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:44.826 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.392 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:44.829 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.396 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:44.845 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.397 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:44.846 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.403 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:44.840 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:18.409 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:44.858 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.527 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:54.975 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.533 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:54.982 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.602 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:55.051 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.605 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:55.055 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.630 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:13:55.078 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.639 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:13:55.088 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.668 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:55.106 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.669 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:55.107 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:28.685 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:13:55.123 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.590 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:05.039 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.595 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:05.044 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.611 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:05.060 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.638 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:05.076 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.639 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:05.077 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.641 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:05.090 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.642 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:05.091 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.656 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:05.094 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:38.660 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:05.109 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.560 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:25.009 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.563 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:25.012 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.596 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:25.045 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.619 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:25.068 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.622 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:25.060 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.624 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:25.073 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.626 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:25.064 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.652 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:25.101 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:46:58.654 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:25.092 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.551 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:29.999 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.553 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:30.002 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.568 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:30.017 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.604 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:30.042 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.605 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:30.043 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.619 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:30.057 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.620 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:30.069 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.626 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:30.073 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:03.642 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:30.091 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.592 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:45.041 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.598 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:45.045 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.612 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:45.061 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.642 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:45.080 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.644 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:45.082 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.645 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:45.095 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.649 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:45.098 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.659 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:45.097 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:18.663 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:45.113 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.571 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:50.020 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.634 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:50.083 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.650 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:14:50.099 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.683 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:50.121 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.687 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:50.125 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.692 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:50.141 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.698 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:50.147 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.699 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:14:50.137 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:23.714 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:14:50.163 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.656 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:05.105 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.658 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:05.095 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.659 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:05.097 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.661 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:05.110 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.681 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:05.118 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.685 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:05.134 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.712 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:05.161 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.713 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:05.163 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:38.733 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:05.182 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.479 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:05.917 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.480 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:05.918 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.491 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:05.929 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.568 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:06.017 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.569 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:06.018 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.578 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:06.027 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.579 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:06.028 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.581 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:06.031 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:39.591 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:06.041 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:53.900 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:20.348 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:53.908 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:20.357 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:53.910 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:20.348 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:53.912 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:20.350 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:53.913 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:20.363 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:53.915 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:20.365 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:54.010 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:20.458 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:54.022 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:20.458 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:54.059 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:20.508 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.781 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:25.230 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.783 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:25.232 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.797 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:25.246 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.827 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:25.264 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.828 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:25.266 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.831 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:25.280 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.833 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:25.283 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.842 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:25.279 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:47:58.846 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:25.295 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.809 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:35.258 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.811 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:35.260 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.829 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:35.278 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.857 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:35.294 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.862 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:35.300 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.862 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:35.311 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.865 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:35.314 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.874 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:35.312 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:08.877 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:35.326 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:16.886 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-12-07T07:48:16.870+0000 25e84640 -1 snap_mapper.add_oid found existing snaps mapped on 219:defdb83f:test-rados-api-smithi097-92795-80::foo:2, removing 2023-12-07T07:48:16.894 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-12-07T07:48:16.885+0000 2e695640 -1 snap_mapper.add_oid found existing snaps mapped on 219:defdb83f:test-rados-api-smithi097-92795-80::foo:2, removing 2023-12-07T07:48:18.989 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:45.437 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:18.993 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:45.442 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:19.018 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:45.465 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:19.062 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:45.511 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:19.064 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:45.513 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:19.064 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:45.502 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:19.077 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:45.515 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:19.079 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:45.529 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:19.092 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:45.530 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.375 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:50.824 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.378 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:50.827 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.398 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:50.847 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.430 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:50.879 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.431 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:50.869 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.431 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:50.880 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.432 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:50.870 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.450 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:50.899 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:24.460 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:50.898 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.874 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:55.323 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.880 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:55.329 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.893 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:15:55.342 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.927 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:55.365 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.927 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:55.376 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.928 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:55.366 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.931 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:55.380 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.941 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:15:55.378 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:28.949 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:15:55.398 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:43.923 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:10.371 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:43.976 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:10.414 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:43.977 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:10.426 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:43.987 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:10.425 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.015 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:10.453 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.031 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:10.480 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.033 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:10.482 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.055 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:10.504 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.066 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:10.516 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.587 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:11.025 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.588 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:11.026 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.596 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:11.034 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.633 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:11.082 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.636 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:11.085 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.646 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:11.096 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.661 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:11.110 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.663 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:11.112 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:44.676 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:11.125 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:55.122 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21619 ms) 2023-12-07T07:48:55.122 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-12-07T07:48:55.122 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25307 ms) 2023-12-07T07:48:55.122 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4409 ms) 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4722 ms) 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (5486 ms) 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4012 ms) 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5685 ms) 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-12-07T07:48:55.123 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39885 ms) 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18682 ms) 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5444 ms) 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4606 ms) 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26260 ms) 2023-12-07T07:48:55.124 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16995 ms) 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6842 ms) 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5781 ms) 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: waiting for scrubs... 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: done waiting 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26392 ms) 2023-12-07T07:48:55.125 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12241 ms) 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10951 ms) 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11385 ms) 2023-12-07T07:48:55.126 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-12-07T07:48:58.995 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:25.444 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:58.999 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:25.449 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:59.031 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:25.480 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:59.043 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:25.481 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:59.044 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:25.482 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:59.046 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:25.495 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:59.050 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:25.498 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:59.073 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:25.510 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:48:59.077 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:25.526 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.238 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:30.687 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.240 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:30.689 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.263 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:30.712 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.300 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:30.738 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.306 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:30.742 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.307 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:30.756 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.310 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:30.759 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.326 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:30.764 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.343 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:30.792 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.848 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:31.297 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.851 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:31.300 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.947 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:31.395 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.991 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:31.440 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:04.995 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:31.445 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:05.008 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:31.457 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:05.040 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:31.477 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:05.042 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:31.480 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:05.134 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:31.572 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.276 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:40.724 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.280 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:40.729 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.314 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:40.763 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.345 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:40.783 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.347 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:40.796 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.352 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:40.790 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.353 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:40.802 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.383 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:40.821 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:14.385 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:40.834 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.436 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:50.885 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.439 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:50.877 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.440 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:50.889 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.441 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:50.879 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.461 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:50.898 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.461 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:50.910 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.486 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:50.935 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.487 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:50.937 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:24.510 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:50.958 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.311 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:55.760 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.313 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:55.762 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.322 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:16:55.771 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.356 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:55.805 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.359 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:55.808 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.366 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:55.803 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.367 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:55.805 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.369 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:16:55.818 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:29.376 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:55.814 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.378 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:15.827 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.379 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:15.829 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.402 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:15.851 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.430 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:15.868 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.431 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:15.869 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.432 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:15.881 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.433 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:15.883 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.453 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:15.891 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:49.456 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:15.905 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.148 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:16.586 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.149 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:16.598 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.149 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:16.587 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.150 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:16.599 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.159 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:16.597 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.160 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:16.610 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.180 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:16.629 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.181 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:16.630 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:49:50.191 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:16.640 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:00.381 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T07:50:00.372+0000 26685640 -1 snap_mapper.add_oid found existing snaps mapped on 77:7faf0ee5:test-rados-api-smithi097-92795-97::foo:31, removing 2023-12-07T07:50:00.392 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-12-07T07:50:00.383+0000 2b68f640 -1 snap_mapper.add_oid found existing snaps mapped on 77:7faf0ee5:test-rados-api-smithi097-92795-97::foo:31, removing 2023-12-07T07:50:04.530 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:30.979 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.532 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:30.981 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.553 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:31.002 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.580 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:31.029 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.581 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:31.030 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.587 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:31.025 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.588 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:31.026 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.599 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:31.049 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:04.606 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:31.044 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.443 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:35.892 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.445 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:35.894 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.458 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:35.907 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.493 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:35.931 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.495 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:35.933 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.496 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:35.945 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.497 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:35.946 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.508 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:35.946 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:09.513 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:35.962 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.588 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:46.037 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.596 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:46.045 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.619 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:46.068 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.640 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:46.077 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.641 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:46.079 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.649 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:46.098 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.650 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:46.100 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.663 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:46.101 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:19.676 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:46.126 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.495 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:55.944 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.498 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:55.947 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.522 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:55.971 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.545 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:55.983 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.547 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:55.997 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.548 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:55.986 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.553 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:56.002 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.570 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:56.008 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:29.576 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:56.025 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.522 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:56.971 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.522 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:56.972 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.535 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:17:56.984 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.554 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:56.992 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.567 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:57.005 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.582 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:57.019 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.739 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:57.188 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.740 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:57.189 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:30.755 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:17:57.204 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.512 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:00.961 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.515 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:00.964 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.534 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:00.983 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.556 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:00.993 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.561 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:01.010 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.562 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:00.999 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.566 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:01.015 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.578 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:01.012 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:34.581 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:01.030 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.588 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:16.036 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.594 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:16.043 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.629 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:16.078 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.646 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:16.095 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.647 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:16.084 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.651 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:16.089 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.653 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:16.102 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.679 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:16.117 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:49.689 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:16.138 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.330 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:16.768 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.331 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:16.769 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.340 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:16.778 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.370 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:16.819 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.371 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:16.820 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.376 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:16.825 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.378 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:16.827 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.380 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:16.830 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:50:50.390 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:16.839 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.726 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:31.174 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.729 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:31.178 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.758 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:31.207 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.776 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:31.225 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.778 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:31.227 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.788 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:31.226 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.793 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:31.231 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.823 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:31.260 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:04.824 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:31.273 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.662 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:36.111 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.664 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:36.113 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.679 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:36.127 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.707 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:36.144 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.709 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:36.147 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.714 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:36.163 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.715 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:36.165 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.722 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:36.160 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:09.729 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:36.179 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.756 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:46.205 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.761 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:46.210 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.788 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:46.237 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.810 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:46.248 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.813 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:46.262 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.813 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:46.251 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.815 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:46.264 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.835 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:46.273 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:19.838 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:46.287 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:29.934 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:56.383 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:29.956 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:56.405 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:29.979 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:18:56.428 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:29.983 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:56.420 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:29.984 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:56.422 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:29.994 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:56.443 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:29.995 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:56.444 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:30.007 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:18:56.445 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:30.018 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:18:56.467 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.845 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:01.294 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.846 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:01.296 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.863 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:01.312 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.905 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:01.342 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.905 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:01.344 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.914 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:01.364 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.916 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:01.365 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.920 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:01.358 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:34.936 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:01.386 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.903 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:16.351 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.905 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:16.354 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.947 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:16.396 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.950 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:16.387 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.952 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:16.390 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.954 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:16.403 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.956 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:16.405 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.972 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:16.410 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:49.985 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:16.434 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.586 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:17.024 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.587 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:17.025 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.595 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:17.033 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.654 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:17.103 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.655 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:17.104 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.655 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:17.104 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.656 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:17.105 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.665 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:17.114 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:51:50.666 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:17.115 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.046 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:31.495 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.051 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:31.500 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.075 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:31.524 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.100 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:31.549 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.100 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:31.538 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.101 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:31.551 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.103 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:31.541 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.125 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:31.563 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:05.125 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:31.574 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.085 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:36.534 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.088 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:36.537 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.111 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:36.560 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.118 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:36.567 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.137 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:36.575 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.141 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:36.579 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.155 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:36.593 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.160 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:36.609 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:10.182 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:36.631 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.026 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:46.474 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.034 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:46.483 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.058 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:46.506 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.078 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:46.515 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.079 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:46.517 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.089 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:46.539 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.092 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:46.541 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.105 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:46.542 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:20.118 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:46.567 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.240 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:51.689 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.242 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:51.692 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.260 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:51.709 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.289 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:51.726 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.294 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:51.730 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.294 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:51.743 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.296 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:51.745 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.309 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:51.747 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:25.314 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:51.763 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.006 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:56.454 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.051 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:56.489 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.065 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:56.503 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.070 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:56.519 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.084 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:19:56.522 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.110 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:19:56.559 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.125 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:56.574 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.128 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:56.577 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:30.143 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:19:56.592 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.218 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:01.667 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.220 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:01.669 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.237 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:01.686 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.263 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:01.712 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.266 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:01.715 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.268 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:01.706 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.269 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:01.707 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.284 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:01.732 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:35.286 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:01.724 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.272 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:16.721 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.331 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:16.780 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.334 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:16.772 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.337 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:16.774 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.370 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:16.808 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.381 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:16.830 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.398 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:16.847 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.401 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:16.851 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:50.431 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:16.880 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:52:53.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:52:53.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11258 ms) 2023-12-07T07:52:53.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-12-07T07:52:53.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:52:53.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7762 ms) 2023-12-07T07:52:53.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-12-07T07:52:53.602 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6705 ms) 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27971 ms) 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15149 ms) 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12901 ms) 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: 00000010 69 73 |is| 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: 00000012 2023-12-07T07:52:53.603 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13195 ms) 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: require_osd_release = reef 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7803 ms) 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (809533 ms total) 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1607 ms) 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10799 ms) 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19106 ms) 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-12-07T07:52:53.604 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (16 ms) 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31528 ms total) 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7979 ms) 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8695 ms) 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: waiting for scrub... 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: done waiting 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26537 ms) 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10420 ms) 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-12-07T07:52:53.605 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (10201 ms) 2023-12-07T07:52:53.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-12-07T07:52:53.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9910 ms) 2023-12-07T07:52:53.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-12-07T07:52:53.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11722 ms) 2023-12-07T07:52:53.606 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-12-07T07:53:05.270 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:31.719 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.273 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:31.722 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.307 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:31.756 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.320 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:31.758 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.321 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:31.759 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.324 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:31.773 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.326 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:31.776 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.349 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:31.786 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:05.358 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:31.806 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.413 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:41.862 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.417 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:41.866 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.453 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:41.902 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.469 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:41.907 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.471 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:41.909 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.488 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:41.937 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.492 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:41.941 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.493 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:41.930 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:15.514 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:41.964 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.349 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:46.798 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.355 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:46.804 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.370 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:20:46.819 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.402 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:46.851 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.405 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:46.854 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.406 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:46.844 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.412 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:46.850 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.419 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:20:46.868 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:20.425 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:20:46.863 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.439 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:01.887 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.443 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:01.891 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.474 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:01.922 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.487 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:01.925 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.490 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:01.928 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.490 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:01.939 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.493 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:01.943 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.519 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:01.957 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:35.522 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:01.971 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.422 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:02.871 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.423 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:02.872 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.437 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:02.884 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.496 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:02.933 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.496 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:02.934 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.510 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:02.948 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.629 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:03.078 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.630 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:03.079 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:36.641 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:03.088 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.638 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:17.087 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.688 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:17.126 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.689 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:17.127 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.707 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:17.156 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.724 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:17.162 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.751 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:17.200 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.756 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:17.206 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.760 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:17.210 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:50.796 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:17.245 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.607 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:22.056 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.608 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:22.058 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.616 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:22.065 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.620 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:22.069 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.633 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:22.082 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.643 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:22.092 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.690 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:22.128 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.694 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:22.132 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:53:55.724 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:22.162 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.624 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:32.072 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.627 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:32.076 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.654 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:32.103 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.678 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:32.126 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.681 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:32.130 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.685 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:32.123 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.687 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:32.125 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.713 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:32.162 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:05.725 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:32.163 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.723 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:42.172 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.775 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:42.213 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.776 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:42.214 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.786 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:42.235 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.799 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:42.237 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.815 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:42.264 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.843 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:42.292 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.844 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:42.294 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:15.863 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:42.312 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.667 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:47.116 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.673 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:47.121 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.682 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:47.131 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.702 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:47.151 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.715 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:47.153 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.718 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:47.156 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.727 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:47.176 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.729 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:47.167 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:20.738 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:47.187 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.389 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:51.838 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.393 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:51.842 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.402 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:51.851 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.403 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:51.841 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.404 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:51.853 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.458 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:51.896 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.461 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:21:51.910 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.467 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:21:51.916 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:25.481 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:21:51.919 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.812 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:02.261 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.815 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:02.264 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.839 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:02.288 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.861 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:02.299 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.862 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:02.300 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.868 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:02.318 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.869 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:02.319 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.885 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:02.323 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:35.894 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:02.343 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.766 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:17.214 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.771 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:17.220 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.799 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:17.248 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.817 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:17.255 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.820 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:17.258 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.821 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:17.271 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.823 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:17.272 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.846 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:17.284 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:50.850 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:17.299 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.782 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:22.230 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.787 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:22.236 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.798 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:22.247 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.834 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:22.283 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.836 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:22.273 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.837 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:22.286 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.838 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:22.275 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.848 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:22.285 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:54:55.848 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:22.297 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.815 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:32.264 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.817 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:32.266 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.835 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:32.284 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.865 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:32.315 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.866 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:32.304 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.867 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:32.305 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.868 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:32.317 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.885 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:32.323 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:05.888 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:32.337 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:15.986 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:42.435 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.007 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:42.455 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.028 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:42.476 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.031 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:42.480 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.033 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:42.482 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.033 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:42.471 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.034 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:42.472 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.055 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:42.504 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:16.056 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:42.494 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.860 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:47.309 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.864 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:47.313 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.876 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:22:47.325 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.917 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:47.355 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.918 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:47.356 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.928 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:22:47.366 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.930 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:47.379 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.933 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:47.382 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:20.944 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:22:47.393 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.150 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:02.599 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.154 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:02.603 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.196 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:02.645 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.216 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:02.653 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.218 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:02.656 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.219 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:02.668 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.224 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:02.673 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.251 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:02.688 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:36.260 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:02.709 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:45.953 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10183 ms) 2023-12-07T07:55:45.953 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-12-07T07:55:45.953 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14521 ms) 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10466 ms) 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15131 ms) 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (10176 ms) 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9267 ms) 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9950 ms) 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-12-07T07:55:45.954 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: first is 1701935652 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,1701935657,1701935658,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,1701935657,1701935658,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,1701935657,1701935658,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,1701935657,1701935658,1701935660,1701935661,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,1701935657,1701935658,1701935660,1701935661,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935652,1701935654,1701935655,1701935657,1701935658,1701935660,1701935661,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: got ls 1701935655,1701935657,1701935658,1701935660,1701935661,1701935663,1701935664,0 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: first now 1701935655, trimmed 2023-12-07T07:55:45.955 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22261 ms) 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: foo0 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (16391 ms) 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20565 ms) 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23272 ms) 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4707 ms) 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4499 ms) 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-12-07T07:55:45.956 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4423 ms) 2023-12-07T07:55:45.957 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-12-07T07:55:45.957 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3389 ms) 2023-12-07T07:55:45.957 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (264666 ms total) 2023-12-07T07:55:45.957 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: 2023-12-07T07:55:45.957 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-12-07T07:55:45.957 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1265667 ms total) 2023-12-07T07:55:45.957 INFO:tasks.workunit.client.0.smithi097.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-12-07T07:55:45.960 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.960 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92547 2023-12-07T07:55:45.960 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92547 2023-12-07T07:55:45.960 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.960 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92706 2023-12-07T07:55:45.960 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92706 2023-12-07T07:55:45.960 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93223 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93223 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92973 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92973 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92786 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92786 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92631 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92631 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92874 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92874 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.961 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92650 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92650 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93141 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93141 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92522 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92522 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92585 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92585 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93181 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93181 2023-12-07T07:55:45.962 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92512 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92512 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92533 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92533 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92720 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92720 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92855 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92855 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92738 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92738 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.963 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93203 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93203 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93079 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93079 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92678 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92678 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93002 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93002 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92505 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92505 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.964 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92564 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92564 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92919 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92919 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=92606 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 92606 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93029 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93029 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ for t in "${!pids[@]}" 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ pid=93068 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ wait 93068 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ exit 0 2023-12-07T07:55:45.965 INFO:tasks.workunit.client.0.smithi097.stderr:+ cleanup 2023-12-07T07:55:45.966 INFO:tasks.workunit.client.0.smithi097.stderr:+ pkill -P 92500 2023-12-07T07:55:45.971 INFO:tasks.workunit.client.0.smithi097.stderr:+ true 2023-12-07T07:55:45.971 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-07T07:55:45.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-07T07:55:46.001 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-12-07T07:55:46.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-07T07:55:46.318 DEBUG:teuthology.parallel:result is None 2023-12-07T07:55:46.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-07T07:55:46.347 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-07T07:55:46.347 DEBUG:teuthology.orchestra.run.smithi097:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-07T07:55:46.410 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-07T07:55:46.411 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-07T07:55:46.422 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-07T07:55:46.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.558+0000 7f924cb84640 10 monclient: get_monmap_and_config 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.558+0000 7f924cb84640 10 monclient: build_initial_monmap 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.558+0000 7f924cb84640 10 monclient: monmap: 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:46.559494+0000 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:46.559494+0000 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:46.562 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.559+0000 7f924cb84640 1 Processor -- start 2023-12-07T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.559+0000 7f924cb84640 1 -- start start 2023-12-07T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.559+0000 7f924cb84640 10 monclient: init 2023-12-07T07:55:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92480af0f0 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af4d0 0x7f9248142040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient(hunting): picked mon.noname-b con 0x7f92480af4d0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9248072760 0x7f92481493d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient(hunting): picked mon.noname-a con 0x7f9248072760 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9248072340 con 0x7f92480af0f0 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9248070b90 con 0x7f9248072760 2023-12-07T07:55:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92480afe80 con 0x7f92480af4d0 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924cb84640 10 monclient(hunting): authenticate will time out at 2668.938645s 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f9245d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af4d0 0x7f9248142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.560+0000 7f924659c640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f92480af0f0 0x7f92480ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:48572/0 (socket says 172.21.15.97:48572) 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.561+0000 7f924659c640 1 -- 172.21.15.97:0/2305298448 learned_addr learned my addr 172.21.15.97:0/2305298448 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.561+0000 7f9245d9b640 10 monclient(hunting): get_auth_request con 0x7f92480af4d0 auth_method 0 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.561+0000 7f9245d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.561+0000 7f9245d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.561+0000 7f9245d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.561+0000 7f9246d9d640 1 --2- 172.21.15.97:0/2305298448 >> v2:172.21.15.97:3300/0 conn(0x7f9248072760 0x7f92481493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.561+0000 7f9245d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9245d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9245d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9246d9d640 10 monclient(hunting): get_auth_request con 0x7f9248072760 auth_method 0 2023-12-07T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9246d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9246d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9246d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9246d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9246d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:46.568 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.562+0000 7f9246d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 10 monclient(hunting): handle_auth_done global_id 17639 payload 274 2023-12-07T07:55:46.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 1 -- 172.21.15.97:0/2305298448 >> v2:172.21.15.97:3300/0 conn(0x7f9248072760 msgr2=0x7f92481493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 1 --2- 172.21.15.97:0/2305298448 >> v2:172.21.15.97:3300/0 conn(0x7f9248072760 0x7f92481493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 1 -- 172.21.15.97:0/2305298448 >> v1:172.21.15.154:6789/0 conn(0x7f92480af0f0 legacy=0x7f92480ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 10 monclient: _finish_hunting 0 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 1 monclient: found mon.noname-b 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.563+0000 7f9245d9b640 1 -- 172.21.15.97:0/2305298448 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924813c600 con 0x7f92480af4d0 2023-12-07T07:55:46.570 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.566+0000 7f9245d9b640 1 --2- 172.21.15.97:0/2305298448 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af4d0 0x7f9248142040 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7f923c0097a0 tx=0x7f923c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=154721620ca523ce server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 1 -- 172.21.15.97:0/2305298448 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c0316f0 con 0x7f92480af4d0 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: dump: 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:46.573 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: _finish_auth 0 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: _check_auth_tickets 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 1 -- 172.21.15.97:0/2305298448 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c031850 con 0x7f92480af4d0 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 1 -- 172.21.15.97:0/2305298448 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c031be0 con 0x7f92480af4d0 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924cb84640 5 monclient: authenticate success, global_id 17639 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924cb84640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924cb84640 4 set_mon_vals no callback set 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.569+0000 7f924559a640 10 monclient: dump: 2023-12-07T07:55:46.574 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 10 monclient: shutdown 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 -- 172.21.15.97:0/2305298448 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af4d0 msgr2=0x7f9248142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 --2- 172.21.15.97:0/2305298448 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af4d0 0x7f9248142040 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7f923c0097a0 tx=0x7f923c02b790 comp rx=0 tx=0).stop 2023-12-07T07:55:46.575 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 -- 172.21.15.97:0/2305298448 shutdown_connections 2023-12-07T07:55:46.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 --2- 172.21.15.97:0/2305298448 >> v2:172.21.15.97:3300/0 conn(0x7f9248072760 0x7f92481493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:46.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 --2- 172.21.15.97:0/2305298448 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af4d0 0x7f9248142040 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:46.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 -- 172.21.15.97:0/2305298448 >> 172.21.15.97:0/2305298448 conn(0x7f92480b01e0 msgr2=0x7f92480b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:46.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 -- 172.21.15.97:0/2305298448 shutdown_connections 2023-12-07T07:55:46.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.570+0000 7f924cb84640 1 -- 172.21.15.97:0/2305298448 wait complete. 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.571+0000 7f924cb84640 10 monclient: build_initial_monmap 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.571+0000 7f924cb84640 10 monclient: monmap: 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:46.579 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.571+0000 7f924cb84640 1 librados: starting msgr at 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.571+0000 7f924cb84640 1 librados: starting objecter 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 Processor -- start 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 -- start start 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 librados: setting wanted keys 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 librados: calling monclient init 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient: init 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:55:46.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9248072760 0x7f9248139350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): picked mon.noname-a con 0x7f9248072760 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af0f0 0x7f9248136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): picked mon.noname-c con 0x7f92480af0f0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9248139a80 0x7f9248137100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): picked mon.noname-b con 0x7f9248139a80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924813b4f0 con 0x7f9248072760 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924813b740 con 0x7f9248139a80 2023-12-07T07:55:46.581 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9248138510 con 0x7f92480af0f0 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.572+0000 7f924cb84640 10 monclient(hunting): authenticate will time out at 2668.950944s 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9248072760 0x7f9248139350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9245d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af0f0 0x7f9248136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9246d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9248139a80 0x7f9248137100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9248072760 0x7f9248139350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38540/0 (socket says 172.21.15.97:38540) 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 1 -- 172.21.15.97:0/3042336198 learned_addr learned my addr 172.21.15.97:0/3042336198 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 10 monclient(hunting): get_auth_request con 0x7f9248072760 auth_method 0 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9245d9b640 10 monclient(hunting): get_auth_request con 0x7f92480af0f0 auth_method 0 2023-12-07T07:55:46.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9245d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9245d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9245d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9246d9d640 10 monclient(hunting): get_auth_request con 0x7f9248139a80 auth_method 0 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9246d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9246d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f9246d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.573+0000 7f924659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.574+0000 7f9246d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.574+0000 7f9246d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.574+0000 7f9246d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.574+0000 7f9245d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.574+0000 7f9245d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:46.583 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.574+0000 7f9245d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 10 monclient(hunting): handle_auth_done global_id 16098 payload 824 2023-12-07T07:55:46.584 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 1 -- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af0f0 msgr2=0x7f9248136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af0f0 0x7f9248136bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 1 -- 172.21.15.97:0/3042336198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9248139a80 msgr2=0x7f9248137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9248139a80 0x7f9248137100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 10 monclient: _finish_hunting 0 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 1 monclient: found mon.noname-a 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.576+0000 7f924659c640 1 -- 172.21.15.97:0/3042336198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c009400 con 0x7f9248072760 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.579+0000 7f924659c640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9248072760 0x7f9248139350 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f923800c4a0 tx=0x7f923800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.579+0000 7f9236ffd640 10 client.?.objecter ms_handle_connect 0x7f9248072760 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.579+0000 7f9236ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 1 -- 172.21.15.97:0/3042336198 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238005ce0 con 0x7f9248072760 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: dump: 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.587 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: _finish_auth 0 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 1 -- 172.21.15.97:0/3042336198 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9238005e40 con 0x7f9248072760 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 5 monclient: authenticate success, global_id 16098 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 10 monclient: _renew_subs 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9248139f10 con 0x7f9248072760 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:46.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 1 -- 172.21.15.97:0/3042336198 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238007270 con 0x7f9248072760 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9236ffd640 10 monclient: dump: 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f9244d99640 4 set_mon_vals no callback set 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 10 client.16098.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:46.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 10 monclient: _renew_subs 2023-12-07T07:55:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924813bb60 con 0x7f9248072760 2023-12-07T07:55:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.582+0000 7f924cb84640 1 librados: init done 2023-12-07T07:55:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.583+0000 7f924cb84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.583+0000 7f924cb84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.583+0000 7f924cb84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:46.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.583+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924813a150 con 0x7f9248072760 2023-12-07T07:55:46.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.826+0000 7f9236ffd640 1 -- 172.21.15.97:0/3042336198 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9238007490 con 0x7f9248072760 2023-12-07T07:55:46.829 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.827+0000 7f9236ffd640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f922806c820 0x7f922806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:46.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.827+0000 7f9245d9b640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f922806c820 0x7f922806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:46.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.827+0000 7f9245d9b640 10 monclient: get_auth_request con 0x7f922806c820 auth_method 0 2023-12-07T07:55:46.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.827+0000 7f9236ffd640 10 client.16098.objecter ms_handle_connect 0x7f922806c820 2023-12-07T07:55:46.830 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.827+0000 7f9245d9b640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f922806c820 0x7f922806ece0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f923c036000 tx=0x7f923c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:46.844 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 1 -- 172.21.15.97:0/3042336198 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f9238057c60 con 0x7f9248072760 2023-12-07T07:55:46.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 10 client.16098.objecter ms_dispatch 0x7f9248000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:46.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 3 client.16098.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:46.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 3 client.16098.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:46.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 20 client.16098.objecter dump_active .. 0 homeless 2023-12-07T07:55:46.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 1 -- 172.21.15.97:0/3042336198 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9238053fa0 con 0x7f9248072760 2023-12-07T07:55:46.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:46.845 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:46.842+0000 7f9236ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:47.027 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.025+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f924813a150 con 0x7f922806c820 2023-12-07T07:55:47.032 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.030+0000 7f9236ffd640 1 -- 172.21.15.97:0/3042336198 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151853 (secure 0 0 0) 0x7f924813a150 con 0x7f922806c820 2023-12-07T07:55:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:55:47.035 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 10 librados: watch_flush enter 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 10 librados: watch_flush exit 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 20 client.16098.objecter shutdown clearing up homeless session... 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 10 client.16098.objecter successfully canceled tick 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f922806c820 msgr2=0x7f922806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f922806c820 0x7f922806ece0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f923c036000 tx=0x7f923c008030 comp rx=0 tx=0).stop 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 10 monclient: shutdown 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9248072760 msgr2=0x7f9248139350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9248072760 0x7f9248139350 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f923800c4a0 tx=0x7f923800c960 comp rx=0 tx=0).stop 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 shutdown_connections 2023-12-07T07:55:47.037 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f922806c820 0x7f922806ece0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9248139a80 0x7f9248137100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f92480af0f0 0x7f9248136bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 --2- 172.21.15.97:0/3042336198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9248072760 0x7f9248139350 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 >> 172.21.15.97:0/3042336198 conn(0x7f92480b01e0 msgr2=0x7f924806e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:47.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 shutdown_connections 2023-12-07T07:55:47.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 -- 172.21.15.97:0/3042336198 wait complete. 2023-12-07T07:55:47.038 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.033+0000 7f924cb84640 1 librados: shutdown 2023-12-07T07:55:47.050 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2023-12-07T07:55:46.061747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542940,"kb_used_data":53360,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754431780,"statfs":{"total":773094113280,"available":772538142720,"internally_reserved":0,"allocated":54640640,"data_stored":30390057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":501206576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":376,"apply_latency_ms":376,"commit_latency_ns":376000000,"apply_latency_ns":376000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.509020"},"pg_stats":[{"pgid":"11.17","version":"0'0","reported_seq":861,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.459982+0000","last_change":"2023-12-07T07:34:49.234367+0000","last_active":"2023-12-07T07:55:40.459982+0000","last_peered":"2023-12-07T07:55:40.459982+0000","last_clean":"2023-12-07T07:55:40.459982+0000","last_became_active":"2023-12-07T07:34:49.221580+0000","last_became_peered":"2023-12-07T07:34:49.221580+0000","last_unstale":"2023-12-07T07:55:40.459982+0000","last_undegraded":"2023-12-07T07:55:40.459982+0000","last_fullsized":"2023-12-07T07:55:40.459982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:19:37.266783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.138320+0000","last_change":"2023-12-07T07:34:47.569420+0000","last_active":"2023-12-07T07:55:43.138320+0000","last_peered":"2023-12-07T07:55:43.138320+0000","last_clean":"2023-12-07T07:55:43.138320+0000","last_became_active":"2023-12-07T07:34:47.563950+0000","last_became_peered":"2023-12-07T07:34:47.563950+0000","last_unstale":"2023-12-07T07:55:43.138320+0000","last_undegraded":"2023-12-07T07:55:43.138320+0000","last_fullsized":"2023-12-07T07:55:43.138320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:43.199927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":874,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.447043+0000","last_change":"2023-12-07T07:34:47.040719+0000","last_active":"2023-12-07T07:55:40.447043+0000","last_peered":"2023-12-07T07:55:40.447043+0000","last_clean":"2023-12-07T07:55:40.447043+0000","last_became_active":"2023-12-07T07:34:47.035305+0000","last_became_peered":"2023-12-07T07:34:47.035305+0000","last_unstale":"2023-12-07T07:55:40.447043+0000","last_undegraded":"2023-12-07T07:55:40.447043+0000","last_fullsized":"2023-12-07T07:55:40.447043+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:31:49.037782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":879,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.090489+0000","last_change":"2023-12-07T07:34:55.054864+0000","last_active":"2023-12-07T07:55:44.090489+0000","last_peered":"2023-12-07T07:55:44.090489+0000","last_clean":"2023-12-07T07:55:44.090489+0000","last_became_active":"2023-12-07T07:34:55.049584+0000","last_became_peered":"2023-12-07T07:34:55.049584+0000","last_unstale":"2023-12-07T07:55:44.090489+0000","last_undegraded":"2023-12-07T07:55:44.090489+0000","last_fullsized":"2023-12-07T07:55:44.090489+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:33:36.960338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.085712+0000","last_change":"2023-12-07T07:35:04.485603+0000","last_active":"2023-12-07T07:55:43.085712+0000","last_peered":"2023-12-07T07:55:43.085712+0000","last_clean":"2023-12-07T07:55:43.085712+0000","last_became_active":"2023-12-07T07:35:04.480195+0000","last_became_peered":"2023-12-07T07:35:04.480195+0000","last_unstale":"2023-12-07T07:55:43.085712+0000","last_undegraded":"2023-12-07T07:55:43.085712+0000","last_fullsized":"2023-12-07T07:55:43.085712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:46:21.908088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.299869+0000","last_change":"2023-12-07T07:34:51.744201+0000","last_active":"2023-12-07T07:55:44.299869+0000","last_peered":"2023-12-07T07:55:44.299869+0000","last_clean":"2023-12-07T07:55:44.299869+0000","last_became_active":"2023-12-07T07:34:51.735923+0000","last_became_peered":"2023-12-07T07:34:51.735923+0000","last_unstale":"2023-12-07T07:55:44.299869+0000","last_undegraded":"2023-12-07T07:55:44.299869+0000","last_fullsized":"2023-12-07T07:55:44.299869+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:23.046164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":886,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.382705+0000","last_change":"2023-12-07T07:34:47.067218+0000","last_active":"2023-12-07T07:55:41.382705+0000","last_peered":"2023-12-07T07:55:41.382705+0000","last_clean":"2023-12-07T07:55:41.382705+0000","last_became_active":"2023-12-07T07:34:47.052848+0000","last_became_peered":"2023-12-07T07:34:47.052848+0000","last_unstale":"2023-12-07T07:55:41.382705+0000","last_undegraded":"2023-12-07T07:55:41.382705+0000","last_fullsized":"2023-12-07T07:55:41.382705+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:34:00.206441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":874,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.325629+0000","last_change":"2023-12-07T07:34:51.124246+0000","last_active":"2023-12-07T07:55:40.325629+0000","last_peered":"2023-12-07T07:55:40.325629+0000","last_clean":"2023-12-07T07:55:40.325629+0000","last_became_active":"2023-12-07T07:34:51.115413+0000","last_became_peered":"2023-12-07T07:34:51.115413+0000","last_unstale":"2023-12-07T07:55:40.325629+0000","last_undegraded":"2023-12-07T07:55:40.325629+0000","last_fullsized":"2023-12-07T07:55:40.325629+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:15:19.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":888,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.438812+0000","last_change":"2023-12-07T07:34:47.170254+0000","last_active":"2023-12-07T07:55:41.438812+0000","last_peered":"2023-12-07T07:55:41.438812+0000","last_clean":"2023-12-07T07:55:41.438812+0000","last_became_active":"2023-12-07T07:34:47.149617+0000","last_became_peered":"2023-12-07T07:34:47.149617+0000","last_unstale":"2023-12-07T07:55:41.438812+0000","last_undegraded":"2023-12-07T07:55:41.438812+0000","last_fullsized":"2023-12-07T07:55:41.438812+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:02:51.144651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.081083+0000","last_change":"2023-12-07T07:34:53.278594+0000","last_active":"2023-12-07T07:55:44.081083+0000","last_peered":"2023-12-07T07:55:44.081083+0000","last_clean":"2023-12-07T07:55:44.081083+0000","last_became_active":"2023-12-07T07:34:53.269724+0000","last_became_peered":"2023-12-07T07:34:53.269724+0000","last_unstale":"2023-12-07T07:55:44.081083+0000","last_undegraded":"2023-12-07T07:55:44.081083+0000","last_fullsized":"2023-12-07T07:55:44.081083+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:30:32.694489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.104600+0000","last_change":"2023-12-07T07:34:56.759465+0000","last_active":"2023-12-07T07:55:43.104600+0000","last_peered":"2023-12-07T07:55:43.104600+0000","last_clean":"2023-12-07T07:55:43.104600+0000","last_became_active":"2023-12-07T07:34:56.746142+0000","last_became_peered":"2023-12-07T07:34:56.746142+0000","last_unstale":"2023-12-07T07:55:43.104600+0000","last_undegraded":"2023-12-07T07:55:43.104600+0000","last_fullsized":"2023-12-07T07:55:43.104600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T16:37:43.750416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.314285+0000","last_change":"2023-12-07T07:34:49.363608+0000","last_active":"2023-12-07T07:55:44.314285+0000","last_peered":"2023-12-07T07:55:44.314285+0000","last_clean":"2023-12-07T07:55:44.314285+0000","last_became_active":"2023-12-07T07:34:49.356256+0000","last_became_peered":"2023-12-07T07:34:49.356256+0000","last_unstale":"2023-12-07T07:55:44.314285+0000","last_undegraded":"2023-12-07T07:55:44.314285+0000","last_fullsized":"2023-12-07T07:55:44.314285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:27:32.615958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.083100+0000","last_change":"2023-12-07T07:33:42.670811+0000","last_active":"2023-12-07T07:55:44.083100+0000","last_peered":"2023-12-07T07:55:44.083100+0000","last_clean":"2023-12-07T07:55:44.083100+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:55:44.083100+0000","last_undegraded":"2023-12-07T07:55:44.083100+0000","last_fullsized":"2023-12-07T07:55:44.083100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:33:38.367199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":862,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.451617+0000","last_change":"2023-12-07T07:34:48.180457+0000","last_active":"2023-12-07T07:55:40.451617+0000","last_peered":"2023-12-07T07:55:40.451617+0000","last_clean":"2023-12-07T07:55:40.451617+0000","last_became_active":"2023-12-07T07:34:48.171478+0000","last_became_peered":"2023-12-07T07:34:48.171478+0000","last_unstale":"2023-12-07T07:55:40.451617+0000","last_undegraded":"2023-12-07T07:55:40.451617+0000","last_fullsized":"2023-12-07T07:55:40.451617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:42.397424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":873,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.351500+0000","last_change":"2023-12-07T07:33:43.216945+0000","last_active":"2023-12-07T07:55:40.351500+0000","last_peered":"2023-12-07T07:55:40.351500+0000","last_clean":"2023-12-07T07:55:40.351500+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:55:40.351500+0000","last_undegraded":"2023-12-07T07:55:40.351500+0000","last_fullsized":"2023-12-07T07:55:40.351500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:54.221713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":862,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.458396+0000","last_change":"2023-12-07T07:34:49.998279+0000","last_active":"2023-12-07T07:55:40.458396+0000","last_peered":"2023-12-07T07:55:40.458396+0000","last_clean":"2023-12-07T07:55:40.458396+0000","last_became_active":"2023-12-07T07:34:49.980936+0000","last_became_peered":"2023-12-07T07:34:49.980936+0000","last_unstale":"2023-12-07T07:55:40.458396+0000","last_undegraded":"2023-12-07T07:55:40.458396+0000","last_fullsized":"2023-12-07T07:55:40.458396+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:00:32.835476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.059682+0000","last_change":"2023-12-07T07:33:42.426643+0000","last_active":"2023-12-07T07:55:44.059682+0000","last_peered":"2023-12-07T07:55:44.059682+0000","last_clean":"2023-12-07T07:55:44.059682+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:55:44.059682+0000","last_undegraded":"2023-12-07T07:55:44.059682+0000","last_fullsized":"2023-12-07T07:55:44.059682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:11:45.155210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":887,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.398030+0000","last_change":"2023-12-07T07:34:51.761228+0000","last_active":"2023-12-07T07:55:41.398030+0000","last_peered":"2023-12-07T07:55:41.398030+0000","last_clean":"2023-12-07T07:55:41.398030+0000","last_became_active":"2023-12-07T07:34:51.754214+0000","last_became_peered":"2023-12-07T07:34:51.754214+0000","last_unstale":"2023-12-07T07:55:41.398030+0000","last_undegraded":"2023-12-07T07:55:41.398030+0000","last_fullsized":"2023-12-07T07:55:41.398030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:05:43.242940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":880,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.451415+0000","last_change":"2023-12-07T07:33:42.656077+0000","last_active":"2023-12-07T07:55:40.451415+0000","last_peered":"2023-12-07T07:55:40.451415+0000","last_clean":"2023-12-07T07:55:40.451415+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:55:40.451415+0000","last_undegraded":"2023-12-07T07:55:40.451415+0000","last_fullsized":"2023-12-07T07:55:40.451415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:05:05.686340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":887,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.413213+0000","last_change":"2023-12-07T07:35:04.576294+0000","last_active":"2023-12-07T07:55:41.413213+0000","last_peered":"2023-12-07T07:55:41.413213+0000","last_clean":"2023-12-07T07:55:41.413213+0000","last_became_active":"2023-12-07T07:35:04.566145+0000","last_became_peered":"2023-12-07T07:35:04.566145+0000","last_unstale":"2023-12-07T07:55:41.413213+0000","last_undegraded":"2023-12-07T07:55:41.413213+0000","last_fullsized":"2023-12-07T07:55:41.413213+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:29.588414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.057960+0000","last_change":"2023-12-07T07:33:42.617026+0000","last_active":"2023-12-07T07:55:44.057960+0000","last_peered":"2023-12-07T07:55:44.057960+0000","last_clean":"2023-12-07T07:55:44.057960+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:55:44.057960+0000","last_undegraded":"2023-12-07T07:55:44.057960+0000","last_fullsized":"2023-12-07T07:55:44.057960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:58.336948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":868,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.451651+0000","last_change":"2023-12-07T07:34:47.445712+0000","last_active":"2023-12-07T07:55:40.451651+0000","last_peered":"2023-12-07T07:55:40.451651+0000","last_clean":"2023-12-07T07:55:40.451651+0000","last_became_active":"2023-12-07T07:34:47.437980+0000","last_became_peered":"2023-12-07T07:34:47.437980+0000","last_unstale":"2023-12-07T07:55:40.451651+0000","last_undegraded":"2023-12-07T07:55:40.451651+0000","last_fullsized":"2023-12-07T07:55:40.451651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:52:30.429862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":892,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.384368+0000","last_change":"2023-12-07T07:33:42.269533+0000","last_active":"2023-12-07T07:55:41.384368+0000","last_peered":"2023-12-07T07:55:41.384368+0000","last_clean":"2023-12-07T07:55:41.384368+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:55:41.384368+0000","last_undegraded":"2023-12-07T07:55:41.384368+0000","last_fullsized":"2023-12-07T07:55:41.384368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:12:50.238470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":868,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.422419+0000","last_change":"2023-12-07T07:35:04.029478+0000","last_active":"2023-12-07T07:55:40.422419+0000","last_peered":"2023-12-07T07:55:40.422419+0000","last_clean":"2023-12-07T07:55:40.422419+0000","last_became_active":"2023-12-07T07:35:04.015545+0000","last_became_peered":"2023-12-07T07:35:04.015545+0000","last_unstale":"2023-12-07T07:55:40.422419+0000","last_undegraded":"2023-12-07T07:55:40.422419+0000","last_fullsized":"2023-12-07T07:55:40.422419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:16:10.231382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.093168+0000","last_change":"2023-12-07T07:33:42.349265+0000","last_active":"2023-12-07T07:55:44.093168+0000","last_peered":"2023-12-07T07:55:44.093168+0000","last_clean":"2023-12-07T07:55:44.093168+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:55:44.093168+0000","last_undegraded":"2023-12-07T07:55:44.093168+0000","last_fullsized":"2023-12-07T07:55:44.093168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:03:37.867596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.328248+0000","last_change":"2023-12-07T07:34:52.478105+0000","last_active":"2023-12-07T07:55:40.328248+0000","last_peered":"2023-12-07T07:55:40.328248+0000","last_clean":"2023-12-07T07:55:40.328248+0000","last_became_active":"2023-12-07T07:34:52.469358+0000","last_became_peered":"2023-12-07T07:34:52.469358+0000","last_unstale":"2023-12-07T07:55:40.328248+0000","last_undegraded":"2023-12-07T07:55:40.328248+0000","last_fullsized":"2023-12-07T07:55:40.328248+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:11:55.093716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.052648+0000","last_change":"2023-12-07T07:33:42.649680+0000","last_active":"2023-12-07T07:55:44.052648+0000","last_peered":"2023-12-07T07:55:44.052648+0000","last_clean":"2023-12-07T07:55:44.052648+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:55:44.052648+0000","last_undegraded":"2023-12-07T07:55:44.052648+0000","last_fullsized":"2023-12-07T07:55:44.052648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:13:55.895630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":976,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.077885+0000","last_change":"2023-12-07T07:33:36.384672+0000","last_active":"2023-12-07T07:55:43.077885+0000","last_peered":"2023-12-07T07:55:43.077885+0000","last_clean":"2023-12-07T07:55:43.077885+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:55:43.077885+0000","last_undegraded":"2023-12-07T07:55:43.077885+0000","last_fullsized":"2023-12-07T07:55:43.077885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_clean_scrub_stamp":"2023-12-07T07:33:30.679434+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:54:14.180629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":875,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.420816+0000","last_change":"2023-12-07T07:34:47.016526+0000","last_active":"2023-12-07T07:55:40.420816+0000","last_peered":"2023-12-07T07:55:40.420816+0000","last_clean":"2023-12-07T07:55:40.420816+0000","last_became_active":"2023-12-07T07:34:47.010508+0000","last_became_peered":"2023-12-07T07:34:47.010508+0000","last_unstale":"2023-12-07T07:55:40.420816+0000","last_undegraded":"2023-12-07T07:55:40.420816+0000","last_fullsized":"2023-12-07T07:55:40.420816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:26:28.206752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.424305+0000","last_change":"2023-12-07T07:35:02.508081+0000","last_active":"2023-12-07T07:55:40.424305+0000","last_peered":"2023-12-07T07:55:40.424305+0000","last_clean":"2023-12-07T07:55:40.424305+0000","last_became_active":"2023-12-07T07:35:02.498406+0000","last_became_peered":"2023-12-07T07:35:02.498406+0000","last_unstale":"2023-12-07T07:55:40.424305+0000","last_undegraded":"2023-12-07T07:55:40.424305+0000","last_fullsized":"2023-12-07T07:55:40.424305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:26:16.927736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1","version":"38'7","reported_seq":878,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.494223+0000","last_change":"2023-12-07T07:34:48.573706+0000","last_active":"2023-12-07T07:55:41.494223+0000","last_peered":"2023-12-07T07:55:41.494223+0000","last_clean":"2023-12-07T07:55:41.494223+0000","last_became_active":"2023-12-07T07:34:48.562400+0000","last_became_peered":"2023-12-07T07:34:48.562400+0000","last_unstale":"2023-12-07T07:55:41.494223+0000","last_undegraded":"2023-12-07T07:55:41.494223+0000","last_fullsized":"2023-12-07T07:55:41.494223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:31:30.011782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":876,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.422434+0000","last_change":"2023-12-07T07:34:50.852474+0000","last_active":"2023-12-07T07:55:40.422434+0000","last_peered":"2023-12-07T07:55:40.422434+0000","last_clean":"2023-12-07T07:55:40.422434+0000","last_became_active":"2023-12-07T07:34:50.845529+0000","last_became_peered":"2023-12-07T07:34:50.845529+0000","last_unstale":"2023-12-07T07:55:40.422434+0000","last_undegraded":"2023-12-07T07:55:40.422434+0000","last_fullsized":"2023-12-07T07:55:40.422434+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:04:08.453958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.049997+0000","last_change":"2023-12-07T07:34:44.988549+0000","last_active":"2023-12-07T07:55:44.049997+0000","last_peered":"2023-12-07T07:55:44.049997+0000","last_clean":"2023-12-07T07:55:44.049997+0000","last_became_active":"2023-12-07T07:34:44.899627+0000","last_became_peered":"2023-12-07T07:34:44.899627+0000","last_unstale":"2023-12-07T07:55:44.049997+0000","last_undegraded":"2023-12-07T07:55:44.049997+0000","last_fullsized":"2023-12-07T07:55:44.049997+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:34:53.809775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":865,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.533937+0000","last_change":"2023-12-07T07:34:50.497421+0000","last_active":"2023-12-07T07:55:41.533937+0000","last_peered":"2023-12-07T07:55:41.533937+0000","last_clean":"2023-12-07T07:55:41.533937+0000","last_became_active":"2023-12-07T07:34:50.418281+0000","last_became_peered":"2023-12-07T07:34:50.418281+0000","last_unstale":"2023-12-07T07:55:41.533937+0000","last_undegraded":"2023-12-07T07:55:41.533937+0000","last_fullsized":"2023-12-07T07:55:41.533937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:26:13.156150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":866,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.106129+0000","last_change":"2023-12-07T07:34:58.289194+0000","last_active":"2023-12-07T07:55:43.106129+0000","last_peered":"2023-12-07T07:55:43.106129+0000","last_clean":"2023-12-07T07:55:43.106129+0000","last_became_active":"2023-12-07T07:34:58.263210+0000","last_became_peered":"2023-12-07T07:34:58.263210+0000","last_unstale":"2023-12-07T07:55:43.106129+0000","last_undegraded":"2023-12-07T07:55:43.106129+0000","last_fullsized":"2023-12-07T07:55:43.106129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:45:56.554479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":875,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.461533+0000","last_change":"2023-12-07T07:34:47.231473+0000","last_active":"2023-12-07T07:55:40.461533+0000","last_peered":"2023-12-07T07:55:40.461533+0000","last_clean":"2023-12-07T07:55:40.461533+0000","last_became_active":"2023-12-07T07:34:47.225369+0000","last_became_peered":"2023-12-07T07:34:47.225369+0000","last_unstale":"2023-12-07T07:55:40.461533+0000","last_undegraded":"2023-12-07T07:55:40.461533+0000","last_fullsized":"2023-12-07T07:55:40.461533+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:53:23.919797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":865,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.164122+0000","last_change":"2023-12-07T07:35:00.370860+0000","last_active":"2023-12-07T07:55:43.164122+0000","last_peered":"2023-12-07T07:55:43.164122+0000","last_clean":"2023-12-07T07:55:43.164122+0000","last_became_active":"2023-12-07T07:35:00.363892+0000","last_became_peered":"2023-12-07T07:35:00.363892+0000","last_unstale":"2023-12-07T07:55:43.164122+0000","last_undegraded":"2023-12-07T07:55:43.164122+0000","last_fullsized":"2023-12-07T07:55:43.164122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:52:59.781747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":867,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.420632+0000","last_change":"2023-12-07T07:34:45.545881+0000","last_active":"2023-12-07T07:55:40.420632+0000","last_peered":"2023-12-07T07:55:40.420632+0000","last_clean":"2023-12-07T07:55:40.420632+0000","last_became_active":"2023-12-07T07:34:45.533595+0000","last_became_peered":"2023-12-07T07:34:45.533595+0000","last_unstale":"2023-12-07T07:55:40.420632+0000","last_undegraded":"2023-12-07T07:55:40.420632+0000","last_fullsized":"2023-12-07T07:55:40.420632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:06:30.943446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":870,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.325599+0000","last_change":"2023-12-07T07:34:54.924870+0000","last_active":"2023-12-07T07:55:40.325599+0000","last_peered":"2023-12-07T07:55:40.325599+0000","last_clean":"2023-12-07T07:55:40.325599+0000","last_became_active":"2023-12-07T07:34:54.918404+0000","last_became_peered":"2023-12-07T07:34:54.918404+0000","last_unstale":"2023-12-07T07:55:40.325599+0000","last_undegraded":"2023-12-07T07:55:40.325599+0000","last_fullsized":"2023-12-07T07:55:40.325599+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:07:29.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":879,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.056428+0000","last_change":"2023-12-07T07:34:45.422416+0000","last_active":"2023-12-07T07:55:44.056428+0000","last_peered":"2023-12-07T07:55:44.056428+0000","last_clean":"2023-12-07T07:55:44.056428+0000","last_became_active":"2023-12-07T07:34:45.408105+0000","last_became_peered":"2023-12-07T07:34:45.408105+0000","last_unstale":"2023-12-07T07:55:44.056428+0000","last_undegraded":"2023-12-07T07:55:44.056428+0000","last_fullsized":"2023-12-07T07:55:44.056428+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:54:33.469967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.309886+0000","last_change":"2023-12-07T07:34:49.403851+0000","last_active":"2023-12-07T07:55:44.309886+0000","last_peered":"2023-12-07T07:55:44.309886+0000","last_clean":"2023-12-07T07:55:44.309886+0000","last_became_active":"2023-12-07T07:34:49.378907+0000","last_became_peered":"2023-12-07T07:34:49.378907+0000","last_unstale":"2023-12-07T07:55:44.309886+0000","last_undegraded":"2023-12-07T07:55:44.309886+0000","last_fullsized":"2023-12-07T07:55:44.309886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:27:11.061093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444299,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77976,"kb_used_data":6540,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293864,"statfs":{"total":96636764160,"available":96556916736,"internally_reserved":0,"allocated":6696960,"data_stored":3657210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":25.919,"15min":28.937999999999999},"min":{"1min":6.7430000000000003,"5min":5.7389999999999999,"15min":3.8220000000000001},"max":{"1min":57.213999999999999,"5min":240.48099999999999,"15min":963.553},"last":100.182},{"interface":"front","average":{"1min":21.895,"5min":32.722000000000001,"15min":26.271999999999998},"min":{"1min":8.7810000000000006,"5min":8.7810000000000006,"15min":4.7640000000000002},"max":{"1min":123.477,"5min":252.37899999999999,"15min":285.09399999999999},"last":23.638999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":9.9960000000000004,"5min":20.056000000000001,"15min":19.806999999999999},"min":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":5.5049999999999999},"max":{"1min":35.368000000000002,"5min":236.608,"15min":240.595},"last":7.4429999999999996},{"interface":"front","average":{"1min":15.894,"5min":28.577000000000002,"15min":25.738},"min":{"1min":9.3689999999999998,"5min":9.0139999999999993,"15min":6.3490000000000002},"max":{"1min":33.170000000000002,"5min":275.33300000000003,"15min":307.64100000000002},"last":8.0999999999999996}]},{"osd":2,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.742000000000001,"5min":29.213999999999999,"15min":23.890999999999998},"min":{"1min":10.449,"5min":9.9049999999999994,"15min":7.4749999999999996},"max":{"1min":56.265000000000001,"5min":277.43099999999998,"15min":277.43099999999998},"last":10.733000000000001},{"interface":"front","average":{"1min":12.913,"5min":22.611999999999998,"15min":24.943000000000001},"min":{"1min":6.3700000000000001,"5min":6.3700000000000001,"15min":6.3700000000000001},"max":{"1min":51.895000000000003,"5min":243.97800000000001,"15min":243.97800000000001},"last":8.7349999999999994}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.201000000000001,"5min":21.512,"15min":22.870000000000001},"min":{"1min":5.7670000000000003,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":37.640999999999998,"5min":195.59299999999999,"15min":314.84300000000002},"last":10.010999999999999},{"interface":"front","average":{"1min":18.228000000000002,"5min":27.143000000000001,"15min":26.853999999999999},"min":{"1min":9.6170000000000009,"5min":7.1500000000000004,"15min":7.1500000000000004},"max":{"1min":66.102999999999994,"5min":215.51900000000001,"15min":252.49799999999999},"last":14.25}]},{"osd":5,"last update":"Thu Dec 7 07:55:22 2023","interfaces":[{"interface":"back","average":{"1min":19.082999999999998,"5min":28.425999999999998,"15min":28.632999999999999},"min":{"1min":7.2640000000000002,"5min":7.2640000000000002,"15min":7.2640000000000002},"max":{"1min":76.856999999999999,"5min":241.82300000000001,"15min":241.82300000000001},"last":13.388},{"interface":"front","average":{"1min":20.713000000000001,"5min":30.940000000000001,"15min":30.433},"min":{"1min":9.3460000000000001,"5min":9.3460000000000001,"15min":8.6780000000000008},"max":{"1min":57.811,"5min":286.17200000000003,"15min":286.17200000000003},"last":22.283000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.530999999999999,"5min":21.175999999999998,"15min":26.318000000000001},"min":{"1min":7.0590000000000002,"5min":6.149,"15min":6.149},"max":{"1min":84.808000000000007,"5min":168.52600000000001,"15min":333.97000000000003},"last":9.2919999999999998},{"interface":"front","average":{"1min":19.992000000000001,"5min":28.367000000000001,"15min":28.050999999999998},"min":{"1min":10.449999999999999,"5min":10.449999999999999,"15min":9.0559999999999992},"max":{"1min":84.061999999999998,"5min":168.88499999999999,"15min":301.428},"last":24.472000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":24.667000000000002,"5min":28.010000000000002,"15min":28.553999999999998},"min":{"1min":12.593999999999999,"5min":8.9139999999999997,"15min":8.0549999999999997},"max":{"1min":83.186000000000007,"5min":186.81299999999999,"15min":247.31100000000001},"last":22.939},{"interface":"front","average":{"1min":19.212,"5min":20.692,"15min":24.864999999999998},"min":{"1min":8.0640000000000001,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":61.241999999999997,"5min":184.453,"15min":311.41000000000003},"last":11.443}]}]},{"osd":2,"up_from":17,"seq":73014444299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64108,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307732,"statfs":{"total":96636764160,"available":96571117568,"internally_reserved":0,"allocated":6709248,"data_stored":3663823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":17.835999999999999,"5min":15.121,"15min":16.925000000000001},"min":{"1min":5.9960000000000004,"5min":5.9960000000000004,"15min":4.9409999999999998},"max":{"1min":111.95099999999999,"5min":149.566,"15min":339.37299999999999},"last":9.9649999999999999},{"interface":"front","average":{"1min":21.946999999999999,"5min":20.587,"15min":23.597999999999999},"min":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":4.6319999999999997},"max":{"1min":123.776,"5min":157.69300000000001,"15min":597.92399999999998},"last":11.215}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.486000000000001,"5min":18.428000000000001,"15min":21.236999999999998},"min":{"1min":7.4930000000000003,"5min":7.4930000000000003,"15min":4.5410000000000004},"max":{"1min":58.052999999999997,"5min":104.455,"15min":282.435},"last":10.6},{"interface":"front","average":{"1min":13.083,"5min":13.429,"15min":18.023},"min":{"1min":5.093,"5min":5.093,"15min":5.093},"max":{"1min":51.125,"5min":100.026,"15min":135.71000000000001},"last":8.7319999999999993}]},{"osd":3,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":21.981999999999999,"5min":23.928000000000001,"15min":20.510999999999999},"min":{"1min":10.015000000000001,"5min":8.1869999999999994,"15min":6.1699999999999999},"max":{"1min":47.886000000000003,"5min":160.94800000000001,"15min":160.94800000000001},"last":15.411},{"interface":"front","average":{"1min":16.581,"5min":18.431000000000001,"15min":21.231000000000002},"min":{"1min":6.5739999999999998,"5min":6.5339999999999998,"15min":6.5170000000000003},"max":{"1min":53.826000000000001,"5min":128.06100000000001,"15min":241.69},"last":8.1769999999999996}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":20.681999999999999,"5min":18.628,"15min":19.465},"min":{"1min":7.5700000000000003,"5min":7.0599999999999996,"15min":6.5570000000000004},"max":{"1min":69.706000000000003,"5min":130.738,"15min":229.14500000000001},"last":28.942},{"interface":"front","average":{"1min":24.361999999999998,"5min":22.751999999999999,"15min":21.934999999999999},"min":{"1min":10.942,"5min":10.942,"15min":5.9829999999999997},"max":{"1min":76.808000000000007,"5min":105.361,"15min":284.95100000000002},"last":25.314}]},{"osd":5,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.340999999999999,"5min":16.795999999999999,"15min":17.544},"min":{"1min":8.1159999999999997,"5min":6.0019999999999998,"15min":5.9489999999999998},"max":{"1min":43.698,"5min":144.864,"15min":247.37799999999999},"last":13.401},{"interface":"front","average":{"1min":18.547999999999998,"5min":21.937999999999999,"15min":21.414999999999999},"min":{"1min":10.209,"5min":6.7229999999999999,"15min":6.7229999999999999},"max":{"1min":57.433999999999997,"5min":143.517,"15min":218.82400000000001},"last":26.041}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":22.513000000000002,"5min":23.238,"15min":25.574000000000002},"min":{"1min":9.5549999999999997,"5min":7.7190000000000003,"15min":7.7190000000000003},"max":{"1min":64.820999999999998,"5min":133.203,"15min":288.38299999999998},"last":23.818999999999999},{"interface":"front","average":{"1min":21.914999999999999,"5min":25.452999999999999,"15min":29.690999999999999},"min":{"1min":12.843999999999999,"5min":10.220000000000001,"15min":8.25},"max":{"1min":55.670999999999999,"5min":182.745,"15min":291.435},"last":23.052}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.765999999999998,"5min":19.440000000000001,"15min":23.538},"min":{"1min":7.7519999999999998,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":45.223999999999997,"5min":162.23500000000001,"15min":302.30900000000003},"last":14.875},{"interface":"front","average":{"1min":22.626000000000001,"5min":25.323,"15min":24.372},"min":{"1min":10.784000000000001,"5min":9.3190000000000008,"15min":5.7859999999999996},"max":{"1min":66.337999999999994,"5min":136.339,"15min":279.00700000000001},"last":27.186}]}]},{"osd":1,"up_from":17,"seq":73014444300,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60328,"kb_used_data":6936,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":7102464,"data_stored":4055055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":14.016,"5min":26.524000000000001,"15min":23.024000000000001},"min":{"1min":4.9160000000000004,"5min":4.3029999999999999,"15min":4.2080000000000002},"max":{"1min":93.808999999999997,"5min":348.17200000000003,"15min":410.08199999999999},"last":8.4000000000000004},{"interface":"front","average":{"1min":15.234999999999999,"5min":20.524999999999999,"15min":20.359000000000002},"min":{"1min":6.4189999999999996,"5min":5.5659999999999998,"15min":5.282},"max":{"1min":82.144999999999996,"5min":146.68700000000001,"15min":303.291},"last":21.623000000000001}]},{"osd":2,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":21.712,"15min":22.257999999999999},"min":{"1min":6.3479999999999999,"5min":4.4240000000000004,"15min":4.4240000000000004},"max":{"1min":23.940000000000001,"5min":192.226,"15min":336.255},"last":9.1639999999999997},{"interface":"front","average":{"1min":18.809999999999999,"5min":28.77,"15min":22.311},"min":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":5.2679999999999998},"max":{"1min":96.281000000000006,"5min":229.56100000000001,"15min":335.88499999999999},"last":21.283000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.492999999999999,"5min":28.306000000000001,"15min":25.484000000000002},"min":{"1min":9.2159999999999993,"5min":7.0449999999999999,"15min":6.3940000000000001},"max":{"1min":87.390000000000001,"5min":317.94999999999999,"15min":429.30399999999997},"last":24.088999999999999},{"interface":"front","average":{"1min":14.365,"5min":31.286000000000001,"15min":24.562000000000001},"min":{"1min":7.1550000000000002,"5min":6.3010000000000002,"15min":6.1589999999999998},"max":{"1min":46.652999999999999,"5min":302.00599999999997,"15min":382.79399999999998},"last":6.6289999999999996}]},{"osd":4,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.879999999999999,"5min":30.443999999999999,"15min":29.187000000000001},"min":{"1min":10.609,"5min":8.7759999999999998,"15min":5.9969999999999999},"max":{"1min":83.418000000000006,"5min":207.262,"15min":406.53899999999999},"last":24.789999999999999},{"interface":"front","average":{"1min":18.018999999999998,"5min":33.978999999999999,"15min":28.32},"min":{"1min":9.6039999999999992,"5min":7.4039999999999999,"15min":6.9279999999999999},"max":{"1min":96.866,"5min":417.81700000000001,"15min":417.81700000000001},"last":20.593}]},{"osd":5,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":23.068999999999999,"5min":27.93,"15min":28.184999999999999},"min":{"1min":11.724,"5min":10.911,"15min":7.3049999999999997},"max":{"1min":106.518,"5min":128.53899999999999,"15min":390.11000000000001},"last":25.827999999999999},{"interface":"front","average":{"1min":21.890000000000001,"5min":33.082000000000001,"15min":27.628},"min":{"1min":10.880000000000001,"5min":8.8249999999999993,"15min":7.0350000000000001},"max":{"1min":102.294,"5min":323.339,"15min":357.44799999999998},"last":22.367000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":22.527000000000001,"5min":29.898,"15min":31.405999999999999},"min":{"1min":11.798,"5min":10.257,"15min":7.9279999999999999},"max":{"1min":90.643000000000001,"5min":188.14400000000001,"15min":420.565},"last":28.507000000000001},{"interface":"front","average":{"1min":26.419,"5min":35.195999999999998,"15min":32.094999999999999},"min":{"1min":14.733000000000001,"5min":12.470000000000001,"15min":8.8680000000000003},"max":{"1min":94.477999999999994,"5min":264.226,"15min":340.71100000000001},"last":54.895000000000003}]},{"osd":7,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":17.436,"5min":24.896999999999998,"15min":27.373999999999999},"min":{"1min":8.2690000000000001,"5min":6.6500000000000004,"15min":6.6500000000000004},"max":{"1min":77.435000000000002,"5min":200.01900000000001,"15min":442.14100000000002},"last":10.077999999999999},{"interface":"front","average":{"1min":24.981000000000002,"5min":32.478999999999999,"15min":30.864000000000001},"min":{"1min":13.936999999999999,"5min":10.494999999999999,"15min":8.8699999999999992},"max":{"1min":98.965999999999994,"5min":267.07999999999998,"15min":533.87199999999996},"last":30.085999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60052,"kb_used_data":6656,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311788,"statfs":{"total":96636764160,"available":96575270912,"internally_reserved":0,"allocated":6815744,"data_stored":3789585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.449,"5min":20.334,"15min":21.199999999999999},"min":{"1min":7.2389999999999999,"5min":5.3540000000000001,"15min":5.3529999999999998},"max":{"1min":52.637,"5min":230.792,"15min":230.792},"last":8.4890000000000008},{"interface":"front","average":{"1min":17.687999999999999,"5min":22.419,"15min":27.146999999999998},"min":{"1min":8.0589999999999993,"5min":6.9210000000000003,"15min":6.9210000000000003},"max":{"1min":68.528000000000006,"5min":119.28,"15min":289.82400000000001},"last":10.679}]},{"osd":2,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":15.130000000000001,"5min":24.141999999999999,"15min":23.670999999999999},"min":{"1min":9.3160000000000007,"5min":8.8759999999999994,"15min":7.9080000000000004},"max":{"1min":26.367000000000001,"5min":201.53800000000001,"15min":201.53800000000001},"last":9.5359999999999996},{"interface":"front","average":{"1min":9.2829999999999995,"5min":17.088999999999999,"15min":16.57},"min":{"1min":5.8230000000000004,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":29.462,"5min":202.648,"15min":214.94},"last":8.7690000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":22.611000000000001,"15min":27.707000000000001},"min":{"1min":6.1550000000000002,"5min":5.3440000000000003,"15min":5.0899999999999999},"max":{"1min":240.876,"5min":283.43000000000001,"15min":932.75999999999999},"last":11.298},{"interface":"front","average":{"1min":20.475000000000001,"5min":26.277999999999999,"15min":27.437000000000001},"min":{"1min":11.481,"5min":8.0809999999999995,"15min":8.0809999999999995},"max":{"1min":78.986999999999995,"5min":256.267,"15min":300.13999999999999},"last":11.99}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":23.826000000000001,"5min":21.247,"15min":28.013000000000002},"min":{"1min":7.1529999999999996,"5min":6.7930000000000001,"15min":6.4649999999999999},"max":{"1min":153.63200000000001,"5min":153.63200000000001,"15min":1049.731},"last":226.23400000000001},{"interface":"front","average":{"1min":27.728000000000002,"5min":27.978999999999999,"15min":28.177},"min":{"1min":13.824,"5min":12.833,"15min":9.5399999999999991},"max":{"1min":108.58799999999999,"5min":158.40100000000001,"15min":283.43700000000001},"last":130.922}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":24.460999999999999,"5min":30.645,"15min":28.616},"min":{"1min":10.550000000000001,"5min":10.550000000000001,"15min":5.8300000000000001},"max":{"1min":105.548,"5min":248.34299999999999,"15min":255.328},"last":53.789000000000001},{"interface":"front","average":{"1min":20.219999999999999,"5min":23.016999999999999,"15min":30.934000000000001},"min":{"1min":6.8079999999999998,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":185.125,"5min":264.34899999999999,"15min":1217.117},"last":108.961}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.291,"5min":21.850000000000001,"15min":31.241},"min":{"1min":8.1869999999999994,"5min":7.8789999999999996,"15min":7.0350000000000001},"max":{"1min":28.216999999999999,"5min":154.68199999999999,"15min":1252.499},"last":13.973000000000001},{"interface":"front","average":{"1min":20.312000000000001,"5min":30.834,"15min":31.664999999999999},"min":{"1min":14.042,"5min":13.167999999999999,"15min":11.638},"max":{"1min":33.158000000000001,"5min":263.43200000000002,"15min":263.43200000000002},"last":13.231999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":22.698,"5min":32.343000000000004,"15min":32.625999999999998},"min":{"1min":15.154999999999999,"5min":13.768000000000001,"15min":13.212},"max":{"1min":37.085000000000001,"5min":140.50200000000001,"15min":279.38900000000001},"last":119.37},{"interface":"front","average":{"1min":14.250999999999999,"5min":24.257999999999999,"15min":23.280999999999999},"min":{"1min":7.8200000000000003,"5min":7.1459999999999999,"15min":6.2990000000000004},"max":{"1min":27.417000000000002,"5min":262.07299999999998,"15min":262.07299999999998},"last":215.11699999999999}]}]},{"osd":6,"up_from":14,"seq":60129542411,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59948,"kb_used_data":6560,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311892,"statfs":{"total":96636764160,"available":96575377408,"internally_reserved":0,"allocated":6717440,"data_stored":3695372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":11.407,"5min":16.442,"15min":18.248999999999999},"min":{"1min":5.4500000000000002,"5min":5.4500000000000002,"15min":4.8579999999999997},"max":{"1min":30.831,"5min":227.49000000000001,"15min":227.49000000000001},"last":59.363999999999997},{"interface":"front","average":{"1min":19.172999999999998,"5min":22.824000000000002,"15min":24.23},"min":{"1min":6.343,"5min":6.343,"15min":5.9909999999999997},"max":{"1min":80.697000000000003,"5min":228.13399999999999,"15min":229.405},"last":69.679000000000002}]},{"osd":1,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":14.680999999999999,"5min":21.905999999999999,"15min":22.215},"min":{"1min":8.3089999999999993,"5min":6.2599999999999998,"15min":5.9390000000000001},"max":{"1min":24.978000000000002,"5min":228.745,"15min":234.59399999999999},"last":52.728000000000002},{"interface":"front","average":{"1min":10.353,"5min":18.015999999999998,"15min":17.309000000000001},"min":{"1min":5.702,"5min":4.8090000000000002,"15min":4.6100000000000003},"max":{"1min":18.276,"5min":220.78800000000001,"15min":220.78800000000001},"last":244.995}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":11.086,"5min":19.494,"15min":18.353000000000002},"min":{"1min":7.4279999999999999,"5min":5.8680000000000003,"15min":4.4210000000000003},"max":{"1min":30.722999999999999,"5min":156.13499999999999,"15min":215.05699999999999},"last":36.252000000000002},{"interface":"front","average":{"1min":16.837,"5min":26.483000000000001,"15min":25.437999999999999},"min":{"1min":10.422000000000001,"5min":7.9950000000000001,"15min":7.9950000000000001},"max":{"1min":26.327000000000002,"5min":232.45400000000001,"15min":232.45400000000001},"last":68.046999999999997}]},{"osd":3,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":12.009,"5min":25.157,"15min":22.254999999999999},"min":{"1min":6.7590000000000003,"5min":6.024,"15min":6.024},"max":{"1min":26.135999999999999,"5min":248.97900000000001,"15min":248.97900000000001},"last":245.643},{"interface":"front","average":{"1min":14.058,"5min":25.068000000000001,"15min":22.209},"min":{"1min":9.5719999999999992,"5min":6.0860000000000003,"15min":6.0860000000000003},"max":{"1min":27.628,"5min":236.92699999999999,"15min":236.92699999999999},"last":56.686}]},{"osd":4,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":19.373999999999999,"5min":27.763000000000002,"15min":23.887},"min":{"1min":6.3899999999999997,"5min":6.3899999999999997,"15min":6.3090000000000002},"max":{"1min":83.894000000000005,"5min":276.92500000000001,"15min":276.92500000000001},"last":246.28299999999999},{"interface":"front","average":{"1min":20.515999999999998,"5min":27.331,"15min":24.074000000000002},"min":{"1min":8.7449999999999992,"5min":7.258,"15min":6.5700000000000003},"max":{"1min":90.427000000000007,"5min":179.64099999999999,"15min":220.881},"last":47.604999999999997}]},{"osd":5,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":16.332000000000001,"5min":26.169,"15min":26.448},"min":{"1min":7.8570000000000002,"5min":7.8570000000000002,"15min":7.8570000000000002},"max":{"1min":34.262999999999998,"5min":237.53700000000001,"15min":237.53700000000001},"last":91.102000000000004},{"interface":"front","average":{"1min":18.193999999999999,"5min":27.972000000000001,"15min":25.896999999999998},"min":{"1min":11.1,"5min":9.1460000000000008,"15min":8.0099999999999998},"max":{"1min":29.687000000000001,"5min":239.673,"15min":239.673},"last":70.638000000000005}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":26.934999999999999,"5min":30.998999999999999,"15min":25.236999999999998},"min":{"1min":10.981999999999999,"5min":8.4890000000000008,"15min":7.3090000000000002},"max":{"1min":107.471,"5min":236.26499999999999,"15min":236.26499999999999},"last":100.85599999999999},{"interface":"front","average":{"1min":28.969999999999999,"5min":31.443999999999999,"15min":28.527999999999999},"min":{"1min":17.210999999999999,"5min":10.239000000000001,"15min":10.239000000000001},"max":{"1min":102.93600000000001,"5min":253.29300000000001,"15min":253.29300000000001},"last":96.189999999999998}]}]},{"osd":7,"up_from":13,"seq":55834575117,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78072,"kb_used_data":6640,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293768,"statfs":{"total":96636764160,"available":96556818432,"internally_reserved":0,"allocated":6799360,"data_stored":3777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":10.605,"5min":11.368,"15min":15.794},"min":{"1min":5.6929999999999996,"5min":4.3899999999999997,"15min":4.29},"max":{"1min":17.640999999999998,"5min":127.59,"15min":214.32599999999999},"last":5.0300000000000002},{"interface":"front","average":{"1min":10.164,"5min":10.188000000000001,"15min":15.063000000000001},"min":{"1min":4.3659999999999997,"5min":4.3659999999999997,"15min":4.3659999999999997},"max":{"1min":20.850000000000001,"5min":47.606000000000002,"15min":199.94499999999999},"last":7.9880000000000004}]},{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":12.381,"5min":16.414000000000001,"15min":19.100000000000001},"min":{"1min":6.3719999999999999,"5min":5.8289999999999997,"15min":5.3849999999999998},"max":{"1min":25.638999999999999,"5min":157.16,"15min":198.512},"last":10.452},{"interface":"front","average":{"1min":12.368,"5min":17.263999999999999,"15min":19.622},"min":{"1min":7.4349999999999996,"5min":5.7270000000000003,"15min":5.6929999999999996},"max":{"1min":18.352,"5min":167.78200000000001,"15min":167.78200000000001},"last":9.125}]},{"osd":2,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.574,"5min":18.367999999999999,"15min":18.265000000000001},"min":{"1min":7.3330000000000002,"5min":6.0049999999999999,"15min":5.9669999999999996},"max":{"1min":23.154,"5min":282.12099999999998,"15min":282.12099999999998},"last":13.823},{"interface":"front","average":{"1min":13.243,"5min":17.161000000000001,"15min":19.664000000000001},"min":{"1min":6.9939999999999998,"5min":6.9939999999999998,"15min":6.9939999999999998},"max":{"1min":20.405000000000001,"5min":154.376,"15min":216.28299999999999},"last":12.545}]},{"osd":3,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":14.811999999999999,"5min":16.736999999999998,"15min":20.478999999999999},"min":{"1min":9.3559999999999999,"5min":6.4939999999999998,"15min":6.3410000000000002},"max":{"1min":21.123999999999999,"5min":84.263000000000005,"15min":159.934},"last":11.082000000000001},{"interface":"front","average":{"1min":15.291,"5min":18.553999999999998,"15min":20.609000000000002},"min":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":6.9409999999999998},"max":{"1min":28.382999999999999,"5min":147.85400000000001,"15min":157.43100000000001},"last":11.829000000000001}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":16.672000000000001,"5min":17.713999999999999,"15min":21.768000000000001},"min":{"1min":9.3829999999999991,"5min":7.2110000000000003,"15min":6.7000000000000002},"max":{"1min":33.926000000000002,"5min":94.227000000000004,"15min":204.71199999999999},"last":13.180999999999999},{"interface":"front","average":{"1min":18.524999999999999,"5min":18.350000000000001,"15min":21.297999999999998},"min":{"1min":9.1869999999999994,"5min":7.7839999999999998,"15min":7.1269999999999998},"max":{"1min":52.731000000000002,"5min":93.356999999999999,"15min":259.23200000000003},"last":16.071999999999999}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.744,"5min":21.847000000000001,"15min":25.094000000000001},"min":{"1min":10.131,"5min":8.9830000000000005,"15min":8.9830000000000005},"max":{"1min":92.340999999999994,"5min":189.49000000000001,"15min":239.32599999999999},"last":16.818000000000001},{"interface":"front","average":{"1min":15.037000000000001,"5min":18.690999999999999,"15min":23.567},"min":{"1min":8.3529999999999998,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":29.849,"5min":157.60400000000001,"15min":224.91900000000001},"last":15.446999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.780999999999999,"5min":20.637,"15min":24.367999999999999},"min":{"1min":12.939,"5min":9.9390000000000001,"15min":8.2750000000000004},"max":{"1min":49.506999999999998,"5min":66.572999999999993,"15min":192.76300000000001},"last":18.07},{"interface":"front","average":{"1min":20.835000000000001,"5min":23.234999999999999,"15min":29.07},"min":{"1min":13.638999999999999,"5min":13.432,"15min":10.176},"max":{"1min":34.186999999999998,"5min":92.688999999999993,"15min":216.917},"last":19.071000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575116,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82524,"kb_used_data":6928,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289316,"statfs":{"total":96636764160,"available":96552259584,"internally_reserved":0,"allocated":7094272,"data_stored":4068636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":11.474,"5min":12.582000000000001,"15min":22.064},"min":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":4.2409999999999997},"max":{"1min":27.518000000000001,"5min":130.309,"15min":1086.095},"last":6.2839999999999998},{"interface":"front","average":{"1min":11.550000000000001,"5min":16.629999999999999,"15min":20.43},"min":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"max":{"1min":38.128999999999998,"5min":141.636,"15min":403.21899999999999},"last":8.7680000000000007}]},{"osd":1,"last update":"Thu Dec 7 07:55:36 2023","interfaces":[{"interface":"back","average":{"1min":20.469000000000001,"5min":15.369,"15min":17.274999999999999},"min":{"1min":6.234,"5min":5.4050000000000002,"15min":4.819},"max":{"1min":130.52500000000001,"5min":130.52500000000001,"15min":310.18299999999999},"last":11.693},{"interface":"front","average":{"1min":23.145,"5min":17.341000000000001,"15min":20.655000000000001},"min":{"1min":7.8479999999999999,"5min":6.4640000000000004,"15min":4.9290000000000003},"max":{"1min":113.23099999999999,"5min":113.23099999999999,"15min":263.452},"last":13.411}]},{"osd":2,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.693000000000001,"5min":19.597000000000001,"15min":21.677},"min":{"1min":6.149,"5min":6.149,"15min":6.149},"max":{"1min":70.923000000000002,"5min":142.77699999999999,"15min":250.30600000000001},"last":14.089},{"interface":"front","average":{"1min":16.704999999999998,"5min":18.449000000000002,"15min":20.291},"min":{"1min":8.2810000000000006,"5min":7.8360000000000003,"15min":6.4059999999999997},"max":{"1min":60.683,"5min":148.40100000000001,"15min":300.988},"last":9.4019999999999992}]},{"osd":3,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":17.314,"5min":22.145,"15min":23.927},"min":{"1min":7.9210000000000003,"5min":7.4370000000000003,"15min":6.9809999999999999},"max":{"1min":55.113999999999997,"5min":143.53899999999999,"15min":305.74900000000002},"last":12.702999999999999},{"interface":"front","average":{"1min":20.635999999999999,"5min":24.911999999999999,"15min":26.052},"min":{"1min":10.500999999999999,"5min":10.500999999999999,"15min":10.500999999999999},"max":{"1min":71.872,"5min":144.41800000000001,"15min":326.85899999999998},"last":17.547999999999998}]},{"osd":4,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":15.747999999999999,"5min":21.678000000000001,"15min":24.283000000000001},"min":{"1min":7.5999999999999996,"5min":6.5899999999999999,"15min":6.5899999999999999},"max":{"1min":68.769999999999996,"5min":136.691,"15min":320.23099999999999},"last":7.3150000000000004},{"interface":"front","average":{"1min":17.957000000000001,"5min":24.734000000000002,"15min":23.757999999999999},"min":{"1min":10.146000000000001,"5min":8.7840000000000007,"15min":6.6929999999999996},"max":{"1min":39.585999999999999,"5min":167.351,"15min":313.42500000000001},"last":15.571999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.943999999999999,"5min":25.558,"15min":30.716999999999999},"min":{"1min":10.445,"5min":9.1029999999999998,"15min":8.9280000000000008},"max":{"1min":35.082999999999998,"5min":201.87299999999999,"15min":320.84199999999998},"last":10.053000000000001},{"interface":"front","average":{"1min":18.722000000000001,"5min":25.786999999999999,"15min":30.187999999999999},"min":{"1min":11.132,"5min":8.3030000000000008,"15min":6.3630000000000004},"max":{"1min":32.395000000000003,"5min":192.81800000000001,"15min":365.91899999999998},"last":18.367999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":13.975,"5min":19.853000000000002,"15min":21.242999999999999},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.484},"max":{"1min":24.908999999999999,"5min":203.636,"15min":286.029},"last":12.359999999999999},{"interface":"front","average":{"1min":17.344999999999999,"5min":24.433,"15min":28.132999999999999},"min":{"1min":8.4990000000000006,"5min":8.4990000000000006,"15min":8.4990000000000006},"max":{"1min":31.541,"5min":204.45500000000001,"15min":300.18200000000002},"last":14.753}]}]},{"osd":4,"up_from":13,"seq":55834575115,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59932,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311908,"statfs":{"total":96636764160,"available":96575393792,"internally_reserved":0,"allocated":6705152,"data_stored":3683084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":10.369999999999999,"5min":13.005000000000001,"15min":17.228999999999999},"min":{"1min":4.5499999999999998,"5min":4.5499999999999998,"15min":4.3230000000000004},"max":{"1min":22.457999999999998,"5min":98.765000000000001,"15min":301.959},"last":6.4450000000000003},{"interface":"front","average":{"1min":11.028,"5min":15.567,"15min":21.140999999999998},"min":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"max":{"1min":24.654,"5min":110.508,"15min":363.75099999999998},"last":7.165}]},{"osd":1,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":14.859999999999999,"5min":15.579000000000001,"15min":20.225000000000001},"min":{"1min":6.4900000000000002,"5min":6.4900000000000002,"15min":5.577},"max":{"1min":57.762,"5min":121.346,"15min":219.19300000000001},"last":12.329000000000001},{"interface":"front","average":{"1min":13.955,"5min":16.913,"15min":20.986000000000001},"min":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3209999999999997},"max":{"1min":38.786000000000001,"5min":111.107,"15min":183.60400000000001},"last":13.404999999999999}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.715999999999999,"5min":21.856000000000002,"15min":20.428999999999998},"min":{"1min":8.4719999999999995,"5min":7.8289999999999997,"15min":6.8710000000000004},"max":{"1min":20.859000000000002,"5min":176.452,"15min":176.452},"last":14.619},{"interface":"front","average":{"1min":15.119999999999999,"5min":22.337,"15min":21.960000000000001},"min":{"1min":8.2330000000000005,"5min":7.6020000000000003,"15min":7.5940000000000003},"max":{"1min":19.547000000000001,"5min":239.935,"15min":239.935},"last":9.7300000000000004}]},{"osd":3,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":9.5579999999999998,"5min":18.57,"15min":22.199000000000002},"min":{"1min":5.968,"5min":5.968,"15min":5.968},"max":{"1min":13.237,"5min":177.48699999999999,"15min":246.523},"last":19.023},{"interface":"front","average":{"1min":14.81,"5min":20.649000000000001,"15min":18.181999999999999},"min":{"1min":7.4720000000000004,"5min":6.274,"15min":5.4240000000000004},"max":{"1min":20.555,"5min":183.75399999999999,"15min":183.75399999999999},"last":21.131}]},{"osd":5,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.641,"5min":15.577,"15min":19.568999999999999},"min":{"1min":8.4770000000000003,"5min":6.327,"15min":6.327},"max":{"1min":20.013000000000002,"5min":123.998,"15min":148.21000000000001},"last":10.952999999999999},{"interface":"front","average":{"1min":18.177,"5min":22.536000000000001,"15min":23.007999999999999},"min":{"1min":13.364000000000001,"5min":9.1270000000000007,"15min":6.46},"max":{"1min":25.591000000000001,"5min":224.55500000000001,"15min":224.55500000000001},"last":12.935}]},{"osd":6,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":13.776999999999999,"5min":20.428000000000001,"15min":23.344000000000001},"min":{"1min":8.4740000000000002,"5min":8.2129999999999992,"15min":7.8079999999999998},"max":{"1min":22.021000000000001,"5min":158.36099999999999,"15min":268.392},"last":15.465999999999999},{"interface":"front","average":{"1min":17.134,"5min":25.366,"15min":26.675999999999998},"min":{"1min":13.542999999999999,"5min":10.300000000000001,"15min":9.9299999999999997},"max":{"1min":23.768000000000001,"5min":243.40700000000001,"15min":243.40700000000001},"last":16.286999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.846,"5min":24.731999999999999,"15min":26.576000000000001},"min":{"1min":10.978,"5min":9.9939999999999998,"15min":9.2319999999999993},"max":{"1min":25.713000000000001,"5min":225.375,"15min":261.75099999999998},"last":17.248000000000001},{"interface":"front","average":{"1min":13.435,"5min":19.010999999999999,"15min":19.09},"min":{"1min":9.1319999999999997,"5min":7.181,"15min":6.7110000000000003},"max":{"1min":20.039000000000001,"5min":167.53700000000001,"15min":178.39699999999999},"last":8.3309999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:55:47.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.150+0000 7fa16396e640 10 monclient: get_monmap_and_config 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.150+0000 7fa16396e640 10 monclient: build_initial_monmap 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.150+0000 7fa16396e640 10 monclient: monmap: 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:47.151702+0000 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:47.151702+0000 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:47.153 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:47.154 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:47.154 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:47.154 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:47.154 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 1 Processor -- start 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 1 -- start start 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient: init 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa15c0a8910 0x7fa15c13ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient(hunting): picked mon.noname-c con 0x7fa15c0a8910 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a9b70 0x7fa15c13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient(hunting): picked mon.noname-b con 0x7fa15c0a9b70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa15c141f90 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0a8cf0 con 0x7fa15c141f90 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c13f6e0 con 0x7fa15c0a8910 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0aabd0 con 0x7fa15c0a9b70 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.151+0000 7fa16396e640 10 monclient(hunting): authenticate will time out at 2669.530272s 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa160f09640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a9b70 0x7fa15c13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa16170a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa15c0a8910 0x7fa15c13ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa160f09640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a9b70 0x7fa15c13f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:59832/0 (socket says 172.21.15.97:59832) 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa160f09640 1 -- 172.21.15.97:0/1448627687 learned_addr learned my addr 172.21.15.97:0/1448627687 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa160f09640 10 monclient(hunting): get_auth_request con 0x7fa15c0a9b70 auth_method 0 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa160f09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa160f09640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa160f09640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa16170a640 10 monclient(hunting): get_auth_request con 0x7fa15c0a8910 auth_method 0 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa16170a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa16170a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.152+0000 7fa16170a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.153+0000 7fa160f09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.153+0000 7fa160f09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.153+0000 7fa160f09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.153+0000 7fa16170a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.153+0000 7fa16170a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.153+0000 7fa16170a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 10 monclient(hunting): handle_auth_done global_id 20164 payload 274 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 1 -- 172.21.15.97:0/1448627687 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a9b70 msgr2=0x7fa15c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 1 --2- 172.21.15.97:0/1448627687 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a9b70 0x7fa15c13f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 1 -- 172.21.15.97:0/1448627687 >> v1:172.21.15.97:6789/0 conn(0x7fa15c141f90 legacy=0x7fa15c142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 10 monclient: _finish_hunting 0 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 1 monclient: found mon.noname-c 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.154+0000 7fa16170a640 1 -- 172.21.15.97:0/1448627687 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0aaeb0 con 0x7fa15c0a8910 2023-12-07T07:55:47.160 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.156+0000 7fa16170a640 1 --2- 172.21.15.97:0/1448627687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0a8910 0x7fa15c13ec60 secure :-1 s=READY pgs=4851 cs=0 l=1 rev1=1 crypto rx=0x7fa1580097a0 tx=0x7fa15802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b637456cf54916 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa153fff640 1 -- 172.21.15.97:0/1448627687 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15802e070 con 0x7fa15c0a8910 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa153fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa153fff640 10 monclient: dump: 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.161 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa153fff640 10 monclient: _finish_auth 0 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa153fff640 10 monclient: _check_auth_tickets 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa153fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa16396e640 5 monclient: authenticate success, global_id 20164 2023-12-07T07:55:47.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.158+0000 7fa16396e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa153fff640 1 -- 172.21.15.97:0/1448627687 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1580315a0 con 0x7fa15c0a8910 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa153fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa153fff640 1 -- 172.21.15.97:0/1448627687 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa158031930 con 0x7fa15c0a8910 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa153fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa153fff640 10 monclient: dump: 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.163 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.164 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.164 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.164 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.164 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.164 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa16396e640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:47.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.160+0000 7fa16396e640 4 set_mon_vals no callback set 2023-12-07T07:55:47.167 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 10 monclient: shutdown 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 -- 172.21.15.97:0/1448627687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0a8910 msgr2=0x7fa15c13ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 --2- 172.21.15.97:0/1448627687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0a8910 0x7fa15c13ec60 secure :-1 s=READY pgs=4851 cs=0 l=1 rev1=1 crypto rx=0x7fa1580097a0 tx=0x7fa15802b840 comp rx=0 tx=0).stop 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 -- 172.21.15.97:0/1448627687 shutdown_connections 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 --2- 172.21.15.97:0/1448627687 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a9b70 0x7fa15c13f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 --2- 172.21.15.97:0/1448627687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0a8910 0x7fa15c13ec60 unknown :-1 s=CLOSED pgs=4851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 -- 172.21.15.97:0/1448627687 >> 172.21.15.97:0/1448627687 conn(0x7fa15c0b7780 msgr2=0x7fa15c0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 -- 172.21.15.97:0/1448627687 shutdown_connections 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.161+0000 7fa16396e640 1 -- 172.21.15.97:0/1448627687 wait complete. 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 10 monclient: build_initial_monmap 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 10 monclient: monmap: 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:47.168 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 1 librados: starting msgr at 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 1 librados: starting objecter 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 1 Processor -- start 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 1 -- start start 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 1 librados: setting wanted keys 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 1 librados: calling monclient init 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.162+0000 7fa16396e640 10 monclient: init 2023-12-07T07:55:47.169 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a8910 0x7fa15c141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): picked mon.noname-c con 0x7fa15c0a8910 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa15c141f90 0x7fa15c13c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): picked mon.noname-a con 0x7fa15c141f90 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0abdc0 0x7fa15c0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): picked mon.noname-b con 0x7fa15c0abdc0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0abc30 con 0x7fa15c141f90 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0aceb0 con 0x7fa15c0abdc0 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c13b240 con 0x7fa15c0a8910 2023-12-07T07:55:47.170 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16396e640 10 monclient(hunting): authenticate will time out at 2669.541588s 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa160f09640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa15c141f90 0x7fa15c13c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa161f0b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0abdc0 0x7fa15c0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.163+0000 7fa16170a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a8910 0x7fa15c141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0abdc0 0x7fa15c0ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:40920/0 (socket says 172.21.15.97:40920) 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 1 -- 172.21.15.97:0/100642748 learned_addr learned my addr 172.21.15.97:0/100642748 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 10 monclient(hunting): get_auth_request con 0x7fa15c0abdc0 auth_method 0 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa160f09640 10 monclient(hunting): get_auth_request con 0x7fa15c141f90 auth_method 0 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa160f09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa160f09640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.171 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa160f09640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa16170a640 10 monclient(hunting): get_auth_request con 0x7fa15c0a8910 auth_method 0 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa16170a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa16170a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa16170a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa161f0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa160f09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa160f09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa160f09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa16170a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa16170a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.172 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.164+0000 7fa16170a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.173 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 10 monclient(hunting): handle_auth_done global_id 20167 payload 824 2023-12-07T07:55:47.173 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 1 -- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a8910 msgr2=0x7fa15c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.173 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a8910 0x7fa15c141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.173 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 1 -- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa15c141f90 msgr2=0x7fa15c13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa15c141f90 0x7fa15c13c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 10 monclient: _finish_hunting 0 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 1 monclient: found mon.noname-b 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.167+0000 7fa161f0b640 1 -- 172.21.15.97:0/100642748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa158009400 con 0x7fa15c0abdc0 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.169+0000 7fa161f0b640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0abdc0 0x7fa15c0ad5b0 secure :-1 s=READY pgs=4852 cs=0 l=1 rev1=1 crypto rx=0x7fa15400b990 tx=0x7fa15400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.169+0000 7fa151ffb640 10 client.?.objecter ms_handle_connect 0x7fa15c0abdc0 2023-12-07T07:55:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.169+0000 7fa151ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 1 -- 172.21.15.97:0/100642748 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15400cba0 con 0x7fa15c0abdc0 2023-12-07T07:55:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: dump: 2023-12-07T07:55:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.175 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: _finish_auth 0 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 1 -- 172.21.15.97:0/100642748 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15400cd00 con 0x7fa15c0abdc0 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 1 -- 172.21.15.97:0/100642748 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154014680 con 0x7fa15c0abdc0 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.176 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa151ffb640 10 monclient: dump: 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa16396e640 5 monclient: authenticate success, global_id 20167 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa16396e640 10 monclient: _renew_subs 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa16396e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c0ab870 con 0x7fa15c0abdc0 2023-12-07T07:55:47.177 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.172+0000 7fa1537fe640 4 set_mon_vals no callback set 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 10 client.20167.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 10 monclient: _renew_subs 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c0adcd0 con 0x7fa15c0abdc0 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 1 librados: init done 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.178 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.173+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15c0aba90 con 0x7fa15c0abdc0 2023-12-07T07:55:47.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.191+0000 7fa151ffb640 1 -- 172.21.15.97:0/100642748 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa15401d3b0 con 0x7fa15c0abdc0 2023-12-07T07:55:47.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.191+0000 7fa151ffb640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa14006c7d0 0x7fa14006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.191+0000 7fa16170a640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa14006c7d0 0x7fa14006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.191+0000 7fa16170a640 10 monclient: get_auth_request con 0x7fa14006c7d0 auth_method 0 2023-12-07T07:55:47.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.192+0000 7fa151ffb640 10 client.20167.objecter ms_handle_connect 0x7fa14006c7d0 2023-12-07T07:55:47.194 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.192+0000 7fa16170a640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa14006c7d0 0x7fa14006ec90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa158036000 tx=0x7fa15803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.194+0000 7fa151ffb640 1 -- 172.21.15.97:0/100642748 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fa1540588f0 con 0x7fa15c0abdc0 2023-12-07T07:55:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.194+0000 7fa151ffb640 10 client.20167.objecter ms_dispatch 0x7fa15c000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.194+0000 7fa151ffb640 3 client.20167.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.194+0000 7fa151ffb640 3 client.20167.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:47.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.194+0000 7fa151ffb640 20 client.20167.objecter dump_active .. 0 homeless 2023-12-07T07:55:47.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.436+0000 7fa151ffb640 1 -- 172.21.15.97:0/100642748 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa154054c30 con 0x7fa15c0abdc0 2023-12-07T07:55:47.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.436+0000 7fa151ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:47.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.436+0000 7fa151ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:47.608 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.606+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa15c0a8fa0 con 0x7fa14006c7d0 2023-12-07T07:55:47.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.610+0000 7fa151ffb640 1 -- 172.21.15.97:0/100642748 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151853 (secure 0 0 0) 0x7fa158031710 con 0x7fa14006c7d0 2023-12-07T07:55:47.614 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:55:47.615 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 10 librados: watch_flush enter 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 10 librados: watch_flush exit 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 20 client.20167.objecter shutdown clearing up homeless session... 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 10 client.20167.objecter successfully canceled tick 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa14006c7d0 msgr2=0x7fa14006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa14006c7d0 0x7fa14006ec90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa158036000 tx=0x7fa15803d040 comp rx=0 tx=0).stop 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 10 monclient: shutdown 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0abdc0 msgr2=0x7fa15c0ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0abdc0 0x7fa15c0ad5b0 secure :-1 s=READY pgs=4852 cs=0 l=1 rev1=1 crypto rx=0x7fa15400b990 tx=0x7fa15400be50 comp rx=0 tx=0).stop 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 shutdown_connections 2023-12-07T07:55:47.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa14006c7d0 0x7fa14006ec90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15c0abdc0 0x7fa15c0ad5b0 unknown :-1 s=CLOSED pgs=4852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa15c141f90 0x7fa15c13c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 --2- 172.21.15.97:0/100642748 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa15c0a8910 0x7fa15c141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 >> 172.21.15.97:0/100642748 conn(0x7fa15c0b7780 msgr2=0x7fa15c0b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:47.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 shutdown_connections 2023-12-07T07:55:47.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 -- 172.21.15.97:0/100642748 wait complete. 2023-12-07T07:55:47.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.613+0000 7fa16396e640 1 librados: shutdown 2023-12-07T07:55:47.627 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2023-12-07T07:55:46.061747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542940,"kb_used_data":53360,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754431780,"statfs":{"total":773094113280,"available":772538142720,"internally_reserved":0,"allocated":54640640,"data_stored":30390057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":501206576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":376,"apply_latency_ms":376,"commit_latency_ns":376000000,"apply_latency_ns":376000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.509020"},"pg_stats":[{"pgid":"11.17","version":"0'0","reported_seq":861,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.459982+0000","last_change":"2023-12-07T07:34:49.234367+0000","last_active":"2023-12-07T07:55:40.459982+0000","last_peered":"2023-12-07T07:55:40.459982+0000","last_clean":"2023-12-07T07:55:40.459982+0000","last_became_active":"2023-12-07T07:34:49.221580+0000","last_became_peered":"2023-12-07T07:34:49.221580+0000","last_unstale":"2023-12-07T07:55:40.459982+0000","last_undegraded":"2023-12-07T07:55:40.459982+0000","last_fullsized":"2023-12-07T07:55:40.459982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:19:37.266783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.138320+0000","last_change":"2023-12-07T07:34:47.569420+0000","last_active":"2023-12-07T07:55:43.138320+0000","last_peered":"2023-12-07T07:55:43.138320+0000","last_clean":"2023-12-07T07:55:43.138320+0000","last_became_active":"2023-12-07T07:34:47.563950+0000","last_became_peered":"2023-12-07T07:34:47.563950+0000","last_unstale":"2023-12-07T07:55:43.138320+0000","last_undegraded":"2023-12-07T07:55:43.138320+0000","last_fullsized":"2023-12-07T07:55:43.138320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:43.199927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":874,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.447043+0000","last_change":"2023-12-07T07:34:47.040719+0000","last_active":"2023-12-07T07:55:40.447043+0000","last_peered":"2023-12-07T07:55:40.447043+0000","last_clean":"2023-12-07T07:55:40.447043+0000","last_became_active":"2023-12-07T07:34:47.035305+0000","last_became_peered":"2023-12-07T07:34:47.035305+0000","last_unstale":"2023-12-07T07:55:40.447043+0000","last_undegraded":"2023-12-07T07:55:40.447043+0000","last_fullsized":"2023-12-07T07:55:40.447043+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:31:49.037782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":879,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.090489+0000","last_change":"2023-12-07T07:34:55.054864+0000","last_active":"2023-12-07T07:55:44.090489+0000","last_peered":"2023-12-07T07:55:44.090489+0000","last_clean":"2023-12-07T07:55:44.090489+0000","last_became_active":"2023-12-07T07:34:55.049584+0000","last_became_peered":"2023-12-07T07:34:55.049584+0000","last_unstale":"2023-12-07T07:55:44.090489+0000","last_undegraded":"2023-12-07T07:55:44.090489+0000","last_fullsized":"2023-12-07T07:55:44.090489+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:33:36.960338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.085712+0000","last_change":"2023-12-07T07:35:04.485603+0000","last_active":"2023-12-07T07:55:43.085712+0000","last_peered":"2023-12-07T07:55:43.085712+0000","last_clean":"2023-12-07T07:55:43.085712+0000","last_became_active":"2023-12-07T07:35:04.480195+0000","last_became_peered":"2023-12-07T07:35:04.480195+0000","last_unstale":"2023-12-07T07:55:43.085712+0000","last_undegraded":"2023-12-07T07:55:43.085712+0000","last_fullsized":"2023-12-07T07:55:43.085712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:46:21.908088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.299869+0000","last_change":"2023-12-07T07:34:51.744201+0000","last_active":"2023-12-07T07:55:44.299869+0000","last_peered":"2023-12-07T07:55:44.299869+0000","last_clean":"2023-12-07T07:55:44.299869+0000","last_became_active":"2023-12-07T07:34:51.735923+0000","last_became_peered":"2023-12-07T07:34:51.735923+0000","last_unstale":"2023-12-07T07:55:44.299869+0000","last_undegraded":"2023-12-07T07:55:44.299869+0000","last_fullsized":"2023-12-07T07:55:44.299869+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:23.046164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":886,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.382705+0000","last_change":"2023-12-07T07:34:47.067218+0000","last_active":"2023-12-07T07:55:41.382705+0000","last_peered":"2023-12-07T07:55:41.382705+0000","last_clean":"2023-12-07T07:55:41.382705+0000","last_became_active":"2023-12-07T07:34:47.052848+0000","last_became_peered":"2023-12-07T07:34:47.052848+0000","last_unstale":"2023-12-07T07:55:41.382705+0000","last_undegraded":"2023-12-07T07:55:41.382705+0000","last_fullsized":"2023-12-07T07:55:41.382705+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:34:00.206441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":874,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.325629+0000","last_change":"2023-12-07T07:34:51.124246+0000","last_active":"2023-12-07T07:55:40.325629+0000","last_peered":"2023-12-07T07:55:40.325629+0000","last_clean":"2023-12-07T07:55:40.325629+0000","last_became_active":"2023-12-07T07:34:51.115413+0000","last_became_peered":"2023-12-07T07:34:51.115413+0000","last_unstale":"2023-12-07T07:55:40.325629+0000","last_undegraded":"2023-12-07T07:55:40.325629+0000","last_fullsized":"2023-12-07T07:55:40.325629+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:15:19.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":888,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.438812+0000","last_change":"2023-12-07T07:34:47.170254+0000","last_active":"2023-12-07T07:55:41.438812+0000","last_peered":"2023-12-07T07:55:41.438812+0000","last_clean":"2023-12-07T07:55:41.438812+0000","last_became_active":"2023-12-07T07:34:47.149617+0000","last_became_peered":"2023-12-07T07:34:47.149617+0000","last_unstale":"2023-12-07T07:55:41.438812+0000","last_undegraded":"2023-12-07T07:55:41.438812+0000","last_fullsized":"2023-12-07T07:55:41.438812+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:02:51.144651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.081083+0000","last_change":"2023-12-07T07:34:53.278594+0000","last_active":"2023-12-07T07:55:44.081083+0000","last_peered":"2023-12-07T07:55:44.081083+0000","last_clean":"2023-12-07T07:55:44.081083+0000","last_became_active":"2023-12-07T07:34:53.269724+0000","last_became_peered":"2023-12-07T07:34:53.269724+0000","last_unstale":"2023-12-07T07:55:44.081083+0000","last_undegraded":"2023-12-07T07:55:44.081083+0000","last_fullsized":"2023-12-07T07:55:44.081083+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:30:32.694489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.104600+0000","last_change":"2023-12-07T07:34:56.759465+0000","last_active":"2023-12-07T07:55:43.104600+0000","last_peered":"2023-12-07T07:55:43.104600+0000","last_clean":"2023-12-07T07:55:43.104600+0000","last_became_active":"2023-12-07T07:34:56.746142+0000","last_became_peered":"2023-12-07T07:34:56.746142+0000","last_unstale":"2023-12-07T07:55:43.104600+0000","last_undegraded":"2023-12-07T07:55:43.104600+0000","last_fullsized":"2023-12-07T07:55:43.104600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T16:37:43.750416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.314285+0000","last_change":"2023-12-07T07:34:49.363608+0000","last_active":"2023-12-07T07:55:44.314285+0000","last_peered":"2023-12-07T07:55:44.314285+0000","last_clean":"2023-12-07T07:55:44.314285+0000","last_became_active":"2023-12-07T07:34:49.356256+0000","last_became_peered":"2023-12-07T07:34:49.356256+0000","last_unstale":"2023-12-07T07:55:44.314285+0000","last_undegraded":"2023-12-07T07:55:44.314285+0000","last_fullsized":"2023-12-07T07:55:44.314285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:27:32.615958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.083100+0000","last_change":"2023-12-07T07:33:42.670811+0000","last_active":"2023-12-07T07:55:44.083100+0000","last_peered":"2023-12-07T07:55:44.083100+0000","last_clean":"2023-12-07T07:55:44.083100+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:55:44.083100+0000","last_undegraded":"2023-12-07T07:55:44.083100+0000","last_fullsized":"2023-12-07T07:55:44.083100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:33:38.367199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":862,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.451617+0000","last_change":"2023-12-07T07:34:48.180457+0000","last_active":"2023-12-07T07:55:40.451617+0000","last_peered":"2023-12-07T07:55:40.451617+0000","last_clean":"2023-12-07T07:55:40.451617+0000","last_became_active":"2023-12-07T07:34:48.171478+0000","last_became_peered":"2023-12-07T07:34:48.171478+0000","last_unstale":"2023-12-07T07:55:40.451617+0000","last_undegraded":"2023-12-07T07:55:40.451617+0000","last_fullsized":"2023-12-07T07:55:40.451617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:42.397424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":873,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.351500+0000","last_change":"2023-12-07T07:33:43.216945+0000","last_active":"2023-12-07T07:55:40.351500+0000","last_peered":"2023-12-07T07:55:40.351500+0000","last_clean":"2023-12-07T07:55:40.351500+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:55:40.351500+0000","last_undegraded":"2023-12-07T07:55:40.351500+0000","last_fullsized":"2023-12-07T07:55:40.351500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:54.221713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":862,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.458396+0000","last_change":"2023-12-07T07:34:49.998279+0000","last_active":"2023-12-07T07:55:40.458396+0000","last_peered":"2023-12-07T07:55:40.458396+0000","last_clean":"2023-12-07T07:55:40.458396+0000","last_became_active":"2023-12-07T07:34:49.980936+0000","last_became_peered":"2023-12-07T07:34:49.980936+0000","last_unstale":"2023-12-07T07:55:40.458396+0000","last_undegraded":"2023-12-07T07:55:40.458396+0000","last_fullsized":"2023-12-07T07:55:40.458396+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:00:32.835476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.059682+0000","last_change":"2023-12-07T07:33:42.426643+0000","last_active":"2023-12-07T07:55:44.059682+0000","last_peered":"2023-12-07T07:55:44.059682+0000","last_clean":"2023-12-07T07:55:44.059682+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:55:44.059682+0000","last_undegraded":"2023-12-07T07:55:44.059682+0000","last_fullsized":"2023-12-07T07:55:44.059682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:11:45.155210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":887,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.398030+0000","last_change":"2023-12-07T07:34:51.761228+0000","last_active":"2023-12-07T07:55:41.398030+0000","last_peered":"2023-12-07T07:55:41.398030+0000","last_clean":"2023-12-07T07:55:41.398030+0000","last_became_active":"2023-12-07T07:34:51.754214+0000","last_became_peered":"2023-12-07T07:34:51.754214+0000","last_unstale":"2023-12-07T07:55:41.398030+0000","last_undegraded":"2023-12-07T07:55:41.398030+0000","last_fullsized":"2023-12-07T07:55:41.398030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:05:43.242940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":880,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.451415+0000","last_change":"2023-12-07T07:33:42.656077+0000","last_active":"2023-12-07T07:55:40.451415+0000","last_peered":"2023-12-07T07:55:40.451415+0000","last_clean":"2023-12-07T07:55:40.451415+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:55:40.451415+0000","last_undegraded":"2023-12-07T07:55:40.451415+0000","last_fullsized":"2023-12-07T07:55:40.451415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:05:05.686340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":887,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.413213+0000","last_change":"2023-12-07T07:35:04.576294+0000","last_active":"2023-12-07T07:55:41.413213+0000","last_peered":"2023-12-07T07:55:41.413213+0000","last_clean":"2023-12-07T07:55:41.413213+0000","last_became_active":"2023-12-07T07:35:04.566145+0000","last_became_peered":"2023-12-07T07:35:04.566145+0000","last_unstale":"2023-12-07T07:55:41.413213+0000","last_undegraded":"2023-12-07T07:55:41.413213+0000","last_fullsized":"2023-12-07T07:55:41.413213+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:29.588414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.057960+0000","last_change":"2023-12-07T07:33:42.617026+0000","last_active":"2023-12-07T07:55:44.057960+0000","last_peered":"2023-12-07T07:55:44.057960+0000","last_clean":"2023-12-07T07:55:44.057960+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:55:44.057960+0000","last_undegraded":"2023-12-07T07:55:44.057960+0000","last_fullsized":"2023-12-07T07:55:44.057960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:58.336948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":868,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.451651+0000","last_change":"2023-12-07T07:34:47.445712+0000","last_active":"2023-12-07T07:55:40.451651+0000","last_peered":"2023-12-07T07:55:40.451651+0000","last_clean":"2023-12-07T07:55:40.451651+0000","last_became_active":"2023-12-07T07:34:47.437980+0000","last_became_peered":"2023-12-07T07:34:47.437980+0000","last_unstale":"2023-12-07T07:55:40.451651+0000","last_undegraded":"2023-12-07T07:55:40.451651+0000","last_fullsized":"2023-12-07T07:55:40.451651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:52:30.429862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":892,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.384368+0000","last_change":"2023-12-07T07:33:42.269533+0000","last_active":"2023-12-07T07:55:41.384368+0000","last_peered":"2023-12-07T07:55:41.384368+0000","last_clean":"2023-12-07T07:55:41.384368+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:55:41.384368+0000","last_undegraded":"2023-12-07T07:55:41.384368+0000","last_fullsized":"2023-12-07T07:55:41.384368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:12:50.238470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":868,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.422419+0000","last_change":"2023-12-07T07:35:04.029478+0000","last_active":"2023-12-07T07:55:40.422419+0000","last_peered":"2023-12-07T07:55:40.422419+0000","last_clean":"2023-12-07T07:55:40.422419+0000","last_became_active":"2023-12-07T07:35:04.015545+0000","last_became_peered":"2023-12-07T07:35:04.015545+0000","last_unstale":"2023-12-07T07:55:40.422419+0000","last_undegraded":"2023-12-07T07:55:40.422419+0000","last_fullsized":"2023-12-07T07:55:40.422419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:16:10.231382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.093168+0000","last_change":"2023-12-07T07:33:42.349265+0000","last_active":"2023-12-07T07:55:44.093168+0000","last_peered":"2023-12-07T07:55:44.093168+0000","last_clean":"2023-12-07T07:55:44.093168+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:55:44.093168+0000","last_undegraded":"2023-12-07T07:55:44.093168+0000","last_fullsized":"2023-12-07T07:55:44.093168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:03:37.867596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":871,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.328248+0000","last_change":"2023-12-07T07:34:52.478105+0000","last_active":"2023-12-07T07:55:40.328248+0000","last_peered":"2023-12-07T07:55:40.328248+0000","last_clean":"2023-12-07T07:55:40.328248+0000","last_became_active":"2023-12-07T07:34:52.469358+0000","last_became_peered":"2023-12-07T07:34:52.469358+0000","last_unstale":"2023-12-07T07:55:40.328248+0000","last_undegraded":"2023-12-07T07:55:40.328248+0000","last_fullsized":"2023-12-07T07:55:40.328248+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:11:55.093716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.052648+0000","last_change":"2023-12-07T07:33:42.649680+0000","last_active":"2023-12-07T07:55:44.052648+0000","last_peered":"2023-12-07T07:55:44.052648+0000","last_clean":"2023-12-07T07:55:44.052648+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:55:44.052648+0000","last_undegraded":"2023-12-07T07:55:44.052648+0000","last_fullsized":"2023-12-07T07:55:44.052648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:13:55.895630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":976,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.077885+0000","last_change":"2023-12-07T07:33:36.384672+0000","last_active":"2023-12-07T07:55:43.077885+0000","last_peered":"2023-12-07T07:55:43.077885+0000","last_clean":"2023-12-07T07:55:43.077885+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:55:43.077885+0000","last_undegraded":"2023-12-07T07:55:43.077885+0000","last_fullsized":"2023-12-07T07:55:43.077885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_clean_scrub_stamp":"2023-12-07T07:33:30.679434+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:54:14.180629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":875,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.420816+0000","last_change":"2023-12-07T07:34:47.016526+0000","last_active":"2023-12-07T07:55:40.420816+0000","last_peered":"2023-12-07T07:55:40.420816+0000","last_clean":"2023-12-07T07:55:40.420816+0000","last_became_active":"2023-12-07T07:34:47.010508+0000","last_became_peered":"2023-12-07T07:34:47.010508+0000","last_unstale":"2023-12-07T07:55:40.420816+0000","last_undegraded":"2023-12-07T07:55:40.420816+0000","last_fullsized":"2023-12-07T07:55:40.420816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:26:28.206752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":869,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.424305+0000","last_change":"2023-12-07T07:35:02.508081+0000","last_active":"2023-12-07T07:55:40.424305+0000","last_peered":"2023-12-07T07:55:40.424305+0000","last_clean":"2023-12-07T07:55:40.424305+0000","last_became_active":"2023-12-07T07:35:02.498406+0000","last_became_peered":"2023-12-07T07:35:02.498406+0000","last_unstale":"2023-12-07T07:55:40.424305+0000","last_undegraded":"2023-12-07T07:55:40.424305+0000","last_fullsized":"2023-12-07T07:55:40.424305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:26:16.927736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1","version":"38'7","reported_seq":878,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.494223+0000","last_change":"2023-12-07T07:34:48.573706+0000","last_active":"2023-12-07T07:55:41.494223+0000","last_peered":"2023-12-07T07:55:41.494223+0000","last_clean":"2023-12-07T07:55:41.494223+0000","last_became_active":"2023-12-07T07:34:48.562400+0000","last_became_peered":"2023-12-07T07:34:48.562400+0000","last_unstale":"2023-12-07T07:55:41.494223+0000","last_undegraded":"2023-12-07T07:55:41.494223+0000","last_fullsized":"2023-12-07T07:55:41.494223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:31:30.011782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":876,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.422434+0000","last_change":"2023-12-07T07:34:50.852474+0000","last_active":"2023-12-07T07:55:40.422434+0000","last_peered":"2023-12-07T07:55:40.422434+0000","last_clean":"2023-12-07T07:55:40.422434+0000","last_became_active":"2023-12-07T07:34:50.845529+0000","last_became_peered":"2023-12-07T07:34:50.845529+0000","last_unstale":"2023-12-07T07:55:40.422434+0000","last_undegraded":"2023-12-07T07:55:40.422434+0000","last_fullsized":"2023-12-07T07:55:40.422434+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:04:08.453958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.049997+0000","last_change":"2023-12-07T07:34:44.988549+0000","last_active":"2023-12-07T07:55:44.049997+0000","last_peered":"2023-12-07T07:55:44.049997+0000","last_clean":"2023-12-07T07:55:44.049997+0000","last_became_active":"2023-12-07T07:34:44.899627+0000","last_became_peered":"2023-12-07T07:34:44.899627+0000","last_unstale":"2023-12-07T07:55:44.049997+0000","last_undegraded":"2023-12-07T07:55:44.049997+0000","last_fullsized":"2023-12-07T07:55:44.049997+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:34:53.809775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":865,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.533937+0000","last_change":"2023-12-07T07:34:50.497421+0000","last_active":"2023-12-07T07:55:41.533937+0000","last_peered":"2023-12-07T07:55:41.533937+0000","last_clean":"2023-12-07T07:55:41.533937+0000","last_became_active":"2023-12-07T07:34:50.418281+0000","last_became_peered":"2023-12-07T07:34:50.418281+0000","last_unstale":"2023-12-07T07:55:41.533937+0000","last_undegraded":"2023-12-07T07:55:41.533937+0000","last_fullsized":"2023-12-07T07:55:41.533937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:26:13.156150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":866,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.106129+0000","last_change":"2023-12-07T07:34:58.289194+0000","last_active":"2023-12-07T07:55:43.106129+0000","last_peered":"2023-12-07T07:55:43.106129+0000","last_clean":"2023-12-07T07:55:43.106129+0000","last_became_active":"2023-12-07T07:34:58.263210+0000","last_became_peered":"2023-12-07T07:34:58.263210+0000","last_unstale":"2023-12-07T07:55:43.106129+0000","last_undegraded":"2023-12-07T07:55:43.106129+0000","last_fullsized":"2023-12-07T07:55:43.106129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:45:56.554479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":875,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.461533+0000","last_change":"2023-12-07T07:34:47.231473+0000","last_active":"2023-12-07T07:55:40.461533+0000","last_peered":"2023-12-07T07:55:40.461533+0000","last_clean":"2023-12-07T07:55:40.461533+0000","last_became_active":"2023-12-07T07:34:47.225369+0000","last_became_peered":"2023-12-07T07:34:47.225369+0000","last_unstale":"2023-12-07T07:55:40.461533+0000","last_undegraded":"2023-12-07T07:55:40.461533+0000","last_fullsized":"2023-12-07T07:55:40.461533+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:53:23.919797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":865,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.164122+0000","last_change":"2023-12-07T07:35:00.370860+0000","last_active":"2023-12-07T07:55:43.164122+0000","last_peered":"2023-12-07T07:55:43.164122+0000","last_clean":"2023-12-07T07:55:43.164122+0000","last_became_active":"2023-12-07T07:35:00.363892+0000","last_became_peered":"2023-12-07T07:35:00.363892+0000","last_unstale":"2023-12-07T07:55:43.164122+0000","last_undegraded":"2023-12-07T07:55:43.164122+0000","last_fullsized":"2023-12-07T07:55:43.164122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:52:59.781747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":867,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.420632+0000","last_change":"2023-12-07T07:34:45.545881+0000","last_active":"2023-12-07T07:55:40.420632+0000","last_peered":"2023-12-07T07:55:40.420632+0000","last_clean":"2023-12-07T07:55:40.420632+0000","last_became_active":"2023-12-07T07:34:45.533595+0000","last_became_peered":"2023-12-07T07:34:45.533595+0000","last_unstale":"2023-12-07T07:55:40.420632+0000","last_undegraded":"2023-12-07T07:55:40.420632+0000","last_fullsized":"2023-12-07T07:55:40.420632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:06:30.943446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":870,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.325599+0000","last_change":"2023-12-07T07:34:54.924870+0000","last_active":"2023-12-07T07:55:40.325599+0000","last_peered":"2023-12-07T07:55:40.325599+0000","last_clean":"2023-12-07T07:55:40.325599+0000","last_became_active":"2023-12-07T07:34:54.918404+0000","last_became_peered":"2023-12-07T07:34:54.918404+0000","last_unstale":"2023-12-07T07:55:40.325599+0000","last_undegraded":"2023-12-07T07:55:40.325599+0000","last_fullsized":"2023-12-07T07:55:40.325599+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:07:29.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":879,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.056428+0000","last_change":"2023-12-07T07:34:45.422416+0000","last_active":"2023-12-07T07:55:44.056428+0000","last_peered":"2023-12-07T07:55:44.056428+0000","last_clean":"2023-12-07T07:55:44.056428+0000","last_became_active":"2023-12-07T07:34:45.408105+0000","last_became_peered":"2023-12-07T07:34:45.408105+0000","last_unstale":"2023-12-07T07:55:44.056428+0000","last_undegraded":"2023-12-07T07:55:44.056428+0000","last_fullsized":"2023-12-07T07:55:44.056428+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:54:33.469967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.309886+0000","last_change":"2023-12-07T07:34:49.403851+0000","last_active":"2023-12-07T07:55:44.309886+0000","last_peered":"2023-12-07T07:55:44.309886+0000","last_clean":"2023-12-07T07:55:44.309886+0000","last_became_active":"2023-12-07T07:34:49.378907+0000","last_became_peered":"2023-12-07T07:34:49.378907+0000","last_unstale":"2023-12-07T07:55:44.309886+0000","last_undegraded":"2023-12-07T07:55:44.309886+0000","last_fullsized":"2023-12-07T07:55:44.309886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:27:11.061093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444299,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77976,"kb_used_data":6540,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293864,"statfs":{"total":96636764160,"available":96556916736,"internally_reserved":0,"allocated":6696960,"data_stored":3657210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":25.919,"15min":28.937999999999999},"min":{"1min":6.7430000000000003,"5min":5.7389999999999999,"15min":3.8220000000000001},"max":{"1min":57.213999999999999,"5min":240.48099999999999,"15min":963.553},"last":100.182},{"interface":"front","average":{"1min":21.895,"5min":32.722000000000001,"15min":26.271999999999998},"min":{"1min":8.7810000000000006,"5min":8.7810000000000006,"15min":4.7640000000000002},"max":{"1min":123.477,"5min":252.37899999999999,"15min":285.09399999999999},"last":23.638999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":9.9960000000000004,"5min":20.056000000000001,"15min":19.806999999999999},"min":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":5.5049999999999999},"max":{"1min":35.368000000000002,"5min":236.608,"15min":240.595},"last":7.4429999999999996},{"interface":"front","average":{"1min":15.894,"5min":28.577000000000002,"15min":25.738},"min":{"1min":9.3689999999999998,"5min":9.0139999999999993,"15min":6.3490000000000002},"max":{"1min":33.170000000000002,"5min":275.33300000000003,"15min":307.64100000000002},"last":8.0999999999999996}]},{"osd":2,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.742000000000001,"5min":29.213999999999999,"15min":23.890999999999998},"min":{"1min":10.449,"5min":9.9049999999999994,"15min":7.4749999999999996},"max":{"1min":56.265000000000001,"5min":277.43099999999998,"15min":277.43099999999998},"last":10.733000000000001},{"interface":"front","average":{"1min":12.913,"5min":22.611999999999998,"15min":24.943000000000001},"min":{"1min":6.3700000000000001,"5min":6.3700000000000001,"15min":6.3700000000000001},"max":{"1min":51.895000000000003,"5min":243.97800000000001,"15min":243.97800000000001},"last":8.7349999999999994}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.201000000000001,"5min":21.512,"15min":22.870000000000001},"min":{"1min":5.7670000000000003,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":37.640999999999998,"5min":195.59299999999999,"15min":314.84300000000002},"last":10.010999999999999},{"interface":"front","average":{"1min":18.228000000000002,"5min":27.143000000000001,"15min":26.853999999999999},"min":{"1min":9.6170000000000009,"5min":7.1500000000000004,"15min":7.1500000000000004},"max":{"1min":66.102999999999994,"5min":215.51900000000001,"15min":252.49799999999999},"last":14.25}]},{"osd":5,"last update":"Thu Dec 7 07:55:22 2023","interfaces":[{"interface":"back","average":{"1min":19.082999999999998,"5min":28.425999999999998,"15min":28.632999999999999},"min":{"1min":7.2640000000000002,"5min":7.2640000000000002,"15min":7.2640000000000002},"max":{"1min":76.856999999999999,"5min":241.82300000000001,"15min":241.82300000000001},"last":13.388},{"interface":"front","average":{"1min":20.713000000000001,"5min":30.940000000000001,"15min":30.433},"min":{"1min":9.3460000000000001,"5min":9.3460000000000001,"15min":8.6780000000000008},"max":{"1min":57.811,"5min":286.17200000000003,"15min":286.17200000000003},"last":22.283000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.530999999999999,"5min":21.175999999999998,"15min":26.318000000000001},"min":{"1min":7.0590000000000002,"5min":6.149,"15min":6.149},"max":{"1min":84.808000000000007,"5min":168.52600000000001,"15min":333.97000000000003},"last":9.2919999999999998},{"interface":"front","average":{"1min":19.992000000000001,"5min":28.367000000000001,"15min":28.050999999999998},"min":{"1min":10.449999999999999,"5min":10.449999999999999,"15min":9.0559999999999992},"max":{"1min":84.061999999999998,"5min":168.88499999999999,"15min":301.428},"last":24.472000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":24.667000000000002,"5min":28.010000000000002,"15min":28.553999999999998},"min":{"1min":12.593999999999999,"5min":8.9139999999999997,"15min":8.0549999999999997},"max":{"1min":83.186000000000007,"5min":186.81299999999999,"15min":247.31100000000001},"last":22.939},{"interface":"front","average":{"1min":19.212,"5min":20.692,"15min":24.864999999999998},"min":{"1min":8.0640000000000001,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":61.241999999999997,"5min":184.453,"15min":311.41000000000003},"last":11.443}]}]},{"osd":2,"up_from":17,"seq":73014444299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64108,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307732,"statfs":{"total":96636764160,"available":96571117568,"internally_reserved":0,"allocated":6709248,"data_stored":3663823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":17.835999999999999,"5min":15.121,"15min":16.925000000000001},"min":{"1min":5.9960000000000004,"5min":5.9960000000000004,"15min":4.9409999999999998},"max":{"1min":111.95099999999999,"5min":149.566,"15min":339.37299999999999},"last":9.9649999999999999},{"interface":"front","average":{"1min":21.946999999999999,"5min":20.587,"15min":23.597999999999999},"min":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":4.6319999999999997},"max":{"1min":123.776,"5min":157.69300000000001,"15min":597.92399999999998},"last":11.215}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.486000000000001,"5min":18.428000000000001,"15min":21.236999999999998},"min":{"1min":7.4930000000000003,"5min":7.4930000000000003,"15min":4.5410000000000004},"max":{"1min":58.052999999999997,"5min":104.455,"15min":282.435},"last":10.6},{"interface":"front","average":{"1min":13.083,"5min":13.429,"15min":18.023},"min":{"1min":5.093,"5min":5.093,"15min":5.093},"max":{"1min":51.125,"5min":100.026,"15min":135.71000000000001},"last":8.7319999999999993}]},{"osd":3,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":21.981999999999999,"5min":23.928000000000001,"15min":20.510999999999999},"min":{"1min":10.015000000000001,"5min":8.1869999999999994,"15min":6.1699999999999999},"max":{"1min":47.886000000000003,"5min":160.94800000000001,"15min":160.94800000000001},"last":15.411},{"interface":"front","average":{"1min":16.581,"5min":18.431000000000001,"15min":21.231000000000002},"min":{"1min":6.5739999999999998,"5min":6.5339999999999998,"15min":6.5170000000000003},"max":{"1min":53.826000000000001,"5min":128.06100000000001,"15min":241.69},"last":8.1769999999999996}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":20.681999999999999,"5min":18.628,"15min":19.465},"min":{"1min":7.5700000000000003,"5min":7.0599999999999996,"15min":6.5570000000000004},"max":{"1min":69.706000000000003,"5min":130.738,"15min":229.14500000000001},"last":28.942},{"interface":"front","average":{"1min":24.361999999999998,"5min":22.751999999999999,"15min":21.934999999999999},"min":{"1min":10.942,"5min":10.942,"15min":5.9829999999999997},"max":{"1min":76.808000000000007,"5min":105.361,"15min":284.95100000000002},"last":25.314}]},{"osd":5,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.340999999999999,"5min":16.795999999999999,"15min":17.544},"min":{"1min":8.1159999999999997,"5min":6.0019999999999998,"15min":5.9489999999999998},"max":{"1min":43.698,"5min":144.864,"15min":247.37799999999999},"last":13.401},{"interface":"front","average":{"1min":18.547999999999998,"5min":21.937999999999999,"15min":21.414999999999999},"min":{"1min":10.209,"5min":6.7229999999999999,"15min":6.7229999999999999},"max":{"1min":57.433999999999997,"5min":143.517,"15min":218.82400000000001},"last":26.041}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":22.513000000000002,"5min":23.238,"15min":25.574000000000002},"min":{"1min":9.5549999999999997,"5min":7.7190000000000003,"15min":7.7190000000000003},"max":{"1min":64.820999999999998,"5min":133.203,"15min":288.38299999999998},"last":23.818999999999999},{"interface":"front","average":{"1min":21.914999999999999,"5min":25.452999999999999,"15min":29.690999999999999},"min":{"1min":12.843999999999999,"5min":10.220000000000001,"15min":8.25},"max":{"1min":55.670999999999999,"5min":182.745,"15min":291.435},"last":23.052}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.765999999999998,"5min":19.440000000000001,"15min":23.538},"min":{"1min":7.7519999999999998,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":45.223999999999997,"5min":162.23500000000001,"15min":302.30900000000003},"last":14.875},{"interface":"front","average":{"1min":22.626000000000001,"5min":25.323,"15min":24.372},"min":{"1min":10.784000000000001,"5min":9.3190000000000008,"15min":5.7859999999999996},"max":{"1min":66.337999999999994,"5min":136.339,"15min":279.00700000000001},"last":27.186}]}]},{"osd":1,"up_from":17,"seq":73014444300,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60328,"kb_used_data":6936,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":7102464,"data_stored":4055055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":14.016,"5min":26.524000000000001,"15min":23.024000000000001},"min":{"1min":4.9160000000000004,"5min":4.3029999999999999,"15min":4.2080000000000002},"max":{"1min":93.808999999999997,"5min":348.17200000000003,"15min":410.08199999999999},"last":8.4000000000000004},{"interface":"front","average":{"1min":15.234999999999999,"5min":20.524999999999999,"15min":20.359000000000002},"min":{"1min":6.4189999999999996,"5min":5.5659999999999998,"15min":5.282},"max":{"1min":82.144999999999996,"5min":146.68700000000001,"15min":303.291},"last":21.623000000000001}]},{"osd":2,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":21.712,"15min":22.257999999999999},"min":{"1min":6.3479999999999999,"5min":4.4240000000000004,"15min":4.4240000000000004},"max":{"1min":23.940000000000001,"5min":192.226,"15min":336.255},"last":9.1639999999999997},{"interface":"front","average":{"1min":18.809999999999999,"5min":28.77,"15min":22.311},"min":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":5.2679999999999998},"max":{"1min":96.281000000000006,"5min":229.56100000000001,"15min":335.88499999999999},"last":21.283000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.492999999999999,"5min":28.306000000000001,"15min":25.484000000000002},"min":{"1min":9.2159999999999993,"5min":7.0449999999999999,"15min":6.3940000000000001},"max":{"1min":87.390000000000001,"5min":317.94999999999999,"15min":429.30399999999997},"last":24.088999999999999},{"interface":"front","average":{"1min":14.365,"5min":31.286000000000001,"15min":24.562000000000001},"min":{"1min":7.1550000000000002,"5min":6.3010000000000002,"15min":6.1589999999999998},"max":{"1min":46.652999999999999,"5min":302.00599999999997,"15min":382.79399999999998},"last":6.6289999999999996}]},{"osd":4,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.879999999999999,"5min":30.443999999999999,"15min":29.187000000000001},"min":{"1min":10.609,"5min":8.7759999999999998,"15min":5.9969999999999999},"max":{"1min":83.418000000000006,"5min":207.262,"15min":406.53899999999999},"last":24.789999999999999},{"interface":"front","average":{"1min":18.018999999999998,"5min":33.978999999999999,"15min":28.32},"min":{"1min":9.6039999999999992,"5min":7.4039999999999999,"15min":6.9279999999999999},"max":{"1min":96.866,"5min":417.81700000000001,"15min":417.81700000000001},"last":20.593}]},{"osd":5,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":23.068999999999999,"5min":27.93,"15min":28.184999999999999},"min":{"1min":11.724,"5min":10.911,"15min":7.3049999999999997},"max":{"1min":106.518,"5min":128.53899999999999,"15min":390.11000000000001},"last":25.827999999999999},{"interface":"front","average":{"1min":21.890000000000001,"5min":33.082000000000001,"15min":27.628},"min":{"1min":10.880000000000001,"5min":8.8249999999999993,"15min":7.0350000000000001},"max":{"1min":102.294,"5min":323.339,"15min":357.44799999999998},"last":22.367000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":22.527000000000001,"5min":29.898,"15min":31.405999999999999},"min":{"1min":11.798,"5min":10.257,"15min":7.9279999999999999},"max":{"1min":90.643000000000001,"5min":188.14400000000001,"15min":420.565},"last":28.507000000000001},{"interface":"front","average":{"1min":26.419,"5min":35.195999999999998,"15min":32.094999999999999},"min":{"1min":14.733000000000001,"5min":12.470000000000001,"15min":8.8680000000000003},"max":{"1min":94.477999999999994,"5min":264.226,"15min":340.71100000000001},"last":54.895000000000003}]},{"osd":7,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":17.436,"5min":24.896999999999998,"15min":27.373999999999999},"min":{"1min":8.2690000000000001,"5min":6.6500000000000004,"15min":6.6500000000000004},"max":{"1min":77.435000000000002,"5min":200.01900000000001,"15min":442.14100000000002},"last":10.077999999999999},{"interface":"front","average":{"1min":24.981000000000002,"5min":32.478999999999999,"15min":30.864000000000001},"min":{"1min":13.936999999999999,"5min":10.494999999999999,"15min":8.8699999999999992},"max":{"1min":98.965999999999994,"5min":267.07999999999998,"15min":533.87199999999996},"last":30.085999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60052,"kb_used_data":6656,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311788,"statfs":{"total":96636764160,"available":96575270912,"internally_reserved":0,"allocated":6815744,"data_stored":3789585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.449,"5min":20.334,"15min":21.199999999999999},"min":{"1min":7.2389999999999999,"5min":5.3540000000000001,"15min":5.3529999999999998},"max":{"1min":52.637,"5min":230.792,"15min":230.792},"last":8.4890000000000008},{"interface":"front","average":{"1min":17.687999999999999,"5min":22.419,"15min":27.146999999999998},"min":{"1min":8.0589999999999993,"5min":6.9210000000000003,"15min":6.9210000000000003},"max":{"1min":68.528000000000006,"5min":119.28,"15min":289.82400000000001},"last":10.679}]},{"osd":2,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":15.130000000000001,"5min":24.141999999999999,"15min":23.670999999999999},"min":{"1min":9.3160000000000007,"5min":8.8759999999999994,"15min":7.9080000000000004},"max":{"1min":26.367000000000001,"5min":201.53800000000001,"15min":201.53800000000001},"last":9.5359999999999996},{"interface":"front","average":{"1min":9.2829999999999995,"5min":17.088999999999999,"15min":16.57},"min":{"1min":5.8230000000000004,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":29.462,"5min":202.648,"15min":214.94},"last":8.7690000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":22.611000000000001,"15min":27.707000000000001},"min":{"1min":6.1550000000000002,"5min":5.3440000000000003,"15min":5.0899999999999999},"max":{"1min":240.876,"5min":283.43000000000001,"15min":932.75999999999999},"last":11.298},{"interface":"front","average":{"1min":20.475000000000001,"5min":26.277999999999999,"15min":27.437000000000001},"min":{"1min":11.481,"5min":8.0809999999999995,"15min":8.0809999999999995},"max":{"1min":78.986999999999995,"5min":256.267,"15min":300.13999999999999},"last":11.99}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":23.826000000000001,"5min":21.247,"15min":28.013000000000002},"min":{"1min":7.1529999999999996,"5min":6.7930000000000001,"15min":6.4649999999999999},"max":{"1min":153.63200000000001,"5min":153.63200000000001,"15min":1049.731},"last":226.23400000000001},{"interface":"front","average":{"1min":27.728000000000002,"5min":27.978999999999999,"15min":28.177},"min":{"1min":13.824,"5min":12.833,"15min":9.5399999999999991},"max":{"1min":108.58799999999999,"5min":158.40100000000001,"15min":283.43700000000001},"last":130.922}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":24.460999999999999,"5min":30.645,"15min":28.616},"min":{"1min":10.550000000000001,"5min":10.550000000000001,"15min":5.8300000000000001},"max":{"1min":105.548,"5min":248.34299999999999,"15min":255.328},"last":53.789000000000001},{"interface":"front","average":{"1min":20.219999999999999,"5min":23.016999999999999,"15min":30.934000000000001},"min":{"1min":6.8079999999999998,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":185.125,"5min":264.34899999999999,"15min":1217.117},"last":108.961}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.291,"5min":21.850000000000001,"15min":31.241},"min":{"1min":8.1869999999999994,"5min":7.8789999999999996,"15min":7.0350000000000001},"max":{"1min":28.216999999999999,"5min":154.68199999999999,"15min":1252.499},"last":13.973000000000001},{"interface":"front","average":{"1min":20.312000000000001,"5min":30.834,"15min":31.664999999999999},"min":{"1min":14.042,"5min":13.167999999999999,"15min":11.638},"max":{"1min":33.158000000000001,"5min":263.43200000000002,"15min":263.43200000000002},"last":13.231999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":22.698,"5min":32.343000000000004,"15min":32.625999999999998},"min":{"1min":15.154999999999999,"5min":13.768000000000001,"15min":13.212},"max":{"1min":37.085000000000001,"5min":140.50200000000001,"15min":279.38900000000001},"last":119.37},{"interface":"front","average":{"1min":14.250999999999999,"5min":24.257999999999999,"15min":23.280999999999999},"min":{"1min":7.8200000000000003,"5min":7.1459999999999999,"15min":6.2990000000000004},"max":{"1min":27.417000000000002,"5min":262.07299999999998,"15min":262.07299999999998},"last":215.11699999999999}]}]},{"osd":6,"up_from":14,"seq":60129542411,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59948,"kb_used_data":6560,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311892,"statfs":{"total":96636764160,"available":96575377408,"internally_reserved":0,"allocated":6717440,"data_stored":3695372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":11.407,"5min":16.442,"15min":18.248999999999999},"min":{"1min":5.4500000000000002,"5min":5.4500000000000002,"15min":4.8579999999999997},"max":{"1min":30.831,"5min":227.49000000000001,"15min":227.49000000000001},"last":59.363999999999997},{"interface":"front","average":{"1min":19.172999999999998,"5min":22.824000000000002,"15min":24.23},"min":{"1min":6.343,"5min":6.343,"15min":5.9909999999999997},"max":{"1min":80.697000000000003,"5min":228.13399999999999,"15min":229.405},"last":69.679000000000002}]},{"osd":1,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":14.680999999999999,"5min":21.905999999999999,"15min":22.215},"min":{"1min":8.3089999999999993,"5min":6.2599999999999998,"15min":5.9390000000000001},"max":{"1min":24.978000000000002,"5min":228.745,"15min":234.59399999999999},"last":52.728000000000002},{"interface":"front","average":{"1min":10.353,"5min":18.015999999999998,"15min":17.309000000000001},"min":{"1min":5.702,"5min":4.8090000000000002,"15min":4.6100000000000003},"max":{"1min":18.276,"5min":220.78800000000001,"15min":220.78800000000001},"last":244.995}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":11.086,"5min":19.494,"15min":18.353000000000002},"min":{"1min":7.4279999999999999,"5min":5.8680000000000003,"15min":4.4210000000000003},"max":{"1min":30.722999999999999,"5min":156.13499999999999,"15min":215.05699999999999},"last":36.252000000000002},{"interface":"front","average":{"1min":16.837,"5min":26.483000000000001,"15min":25.437999999999999},"min":{"1min":10.422000000000001,"5min":7.9950000000000001,"15min":7.9950000000000001},"max":{"1min":26.327000000000002,"5min":232.45400000000001,"15min":232.45400000000001},"last":68.046999999999997}]},{"osd":3,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":12.009,"5min":25.157,"15min":22.254999999999999},"min":{"1min":6.7590000000000003,"5min":6.024,"15min":6.024},"max":{"1min":26.135999999999999,"5min":248.97900000000001,"15min":248.97900000000001},"last":245.643},{"interface":"front","average":{"1min":14.058,"5min":25.068000000000001,"15min":22.209},"min":{"1min":9.5719999999999992,"5min":6.0860000000000003,"15min":6.0860000000000003},"max":{"1min":27.628,"5min":236.92699999999999,"15min":236.92699999999999},"last":56.686}]},{"osd":4,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":19.373999999999999,"5min":27.763000000000002,"15min":23.887},"min":{"1min":6.3899999999999997,"5min":6.3899999999999997,"15min":6.3090000000000002},"max":{"1min":83.894000000000005,"5min":276.92500000000001,"15min":276.92500000000001},"last":246.28299999999999},{"interface":"front","average":{"1min":20.515999999999998,"5min":27.331,"15min":24.074000000000002},"min":{"1min":8.7449999999999992,"5min":7.258,"15min":6.5700000000000003},"max":{"1min":90.427000000000007,"5min":179.64099999999999,"15min":220.881},"last":47.604999999999997}]},{"osd":5,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":16.332000000000001,"5min":26.169,"15min":26.448},"min":{"1min":7.8570000000000002,"5min":7.8570000000000002,"15min":7.8570000000000002},"max":{"1min":34.262999999999998,"5min":237.53700000000001,"15min":237.53700000000001},"last":91.102000000000004},{"interface":"front","average":{"1min":18.193999999999999,"5min":27.972000000000001,"15min":25.896999999999998},"min":{"1min":11.1,"5min":9.1460000000000008,"15min":8.0099999999999998},"max":{"1min":29.687000000000001,"5min":239.673,"15min":239.673},"last":70.638000000000005}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":26.934999999999999,"5min":30.998999999999999,"15min":25.236999999999998},"min":{"1min":10.981999999999999,"5min":8.4890000000000008,"15min":7.3090000000000002},"max":{"1min":107.471,"5min":236.26499999999999,"15min":236.26499999999999},"last":100.85599999999999},{"interface":"front","average":{"1min":28.969999999999999,"5min":31.443999999999999,"15min":28.527999999999999},"min":{"1min":17.210999999999999,"5min":10.239000000000001,"15min":10.239000000000001},"max":{"1min":102.93600000000001,"5min":253.29300000000001,"15min":253.29300000000001},"last":96.189999999999998}]}]},{"osd":7,"up_from":13,"seq":55834575117,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78072,"kb_used_data":6640,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293768,"statfs":{"total":96636764160,"available":96556818432,"internally_reserved":0,"allocated":6799360,"data_stored":3777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":10.605,"5min":11.368,"15min":15.794},"min":{"1min":5.6929999999999996,"5min":4.3899999999999997,"15min":4.29},"max":{"1min":17.640999999999998,"5min":127.59,"15min":214.32599999999999},"last":5.0300000000000002},{"interface":"front","average":{"1min":10.164,"5min":10.188000000000001,"15min":15.063000000000001},"min":{"1min":4.3659999999999997,"5min":4.3659999999999997,"15min":4.3659999999999997},"max":{"1min":20.850000000000001,"5min":47.606000000000002,"15min":199.94499999999999},"last":7.9880000000000004}]},{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":12.381,"5min":16.414000000000001,"15min":19.100000000000001},"min":{"1min":6.3719999999999999,"5min":5.8289999999999997,"15min":5.3849999999999998},"max":{"1min":25.638999999999999,"5min":157.16,"15min":198.512},"last":10.452},{"interface":"front","average":{"1min":12.368,"5min":17.263999999999999,"15min":19.622},"min":{"1min":7.4349999999999996,"5min":5.7270000000000003,"15min":5.6929999999999996},"max":{"1min":18.352,"5min":167.78200000000001,"15min":167.78200000000001},"last":9.125}]},{"osd":2,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.574,"5min":18.367999999999999,"15min":18.265000000000001},"min":{"1min":7.3330000000000002,"5min":6.0049999999999999,"15min":5.9669999999999996},"max":{"1min":23.154,"5min":282.12099999999998,"15min":282.12099999999998},"last":13.823},{"interface":"front","average":{"1min":13.243,"5min":17.161000000000001,"15min":19.664000000000001},"min":{"1min":6.9939999999999998,"5min":6.9939999999999998,"15min":6.9939999999999998},"max":{"1min":20.405000000000001,"5min":154.376,"15min":216.28299999999999},"last":12.545}]},{"osd":3,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":14.811999999999999,"5min":16.736999999999998,"15min":20.478999999999999},"min":{"1min":9.3559999999999999,"5min":6.4939999999999998,"15min":6.3410000000000002},"max":{"1min":21.123999999999999,"5min":84.263000000000005,"15min":159.934},"last":11.082000000000001},{"interface":"front","average":{"1min":15.291,"5min":18.553999999999998,"15min":20.609000000000002},"min":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":6.9409999999999998},"max":{"1min":28.382999999999999,"5min":147.85400000000001,"15min":157.43100000000001},"last":11.829000000000001}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":16.672000000000001,"5min":17.713999999999999,"15min":21.768000000000001},"min":{"1min":9.3829999999999991,"5min":7.2110000000000003,"15min":6.7000000000000002},"max":{"1min":33.926000000000002,"5min":94.227000000000004,"15min":204.71199999999999},"last":13.180999999999999},{"interface":"front","average":{"1min":18.524999999999999,"5min":18.350000000000001,"15min":21.297999999999998},"min":{"1min":9.1869999999999994,"5min":7.7839999999999998,"15min":7.1269999999999998},"max":{"1min":52.731000000000002,"5min":93.356999999999999,"15min":259.23200000000003},"last":16.071999999999999}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.744,"5min":21.847000000000001,"15min":25.094000000000001},"min":{"1min":10.131,"5min":8.9830000000000005,"15min":8.9830000000000005},"max":{"1min":92.340999999999994,"5min":189.49000000000001,"15min":239.32599999999999},"last":16.818000000000001},{"interface":"front","average":{"1min":15.037000000000001,"5min":18.690999999999999,"15min":23.567},"min":{"1min":8.3529999999999998,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":29.849,"5min":157.60400000000001,"15min":224.91900000000001},"last":15.446999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.780999999999999,"5min":20.637,"15min":24.367999999999999},"min":{"1min":12.939,"5min":9.9390000000000001,"15min":8.2750000000000004},"max":{"1min":49.506999999999998,"5min":66.572999999999993,"15min":192.76300000000001},"last":18.07},{"interface":"front","average":{"1min":20.835000000000001,"5min":23.234999999999999,"15min":29.07},"min":{"1min":13.638999999999999,"5min":13.432,"15min":10.176},"max":{"1min":34.186999999999998,"5min":92.688999999999993,"15min":216.917},"last":19.071000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575116,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82524,"kb_used_data":6928,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289316,"statfs":{"total":96636764160,"available":96552259584,"internally_reserved":0,"allocated":7094272,"data_stored":4068636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":11.474,"5min":12.582000000000001,"15min":22.064},"min":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":4.2409999999999997},"max":{"1min":27.518000000000001,"5min":130.309,"15min":1086.095},"last":6.2839999999999998},{"interface":"front","average":{"1min":11.550000000000001,"5min":16.629999999999999,"15min":20.43},"min":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"max":{"1min":38.128999999999998,"5min":141.636,"15min":403.21899999999999},"last":8.7680000000000007}]},{"osd":1,"last update":"Thu Dec 7 07:55:36 2023","interfaces":[{"interface":"back","average":{"1min":20.469000000000001,"5min":15.369,"15min":17.274999999999999},"min":{"1min":6.234,"5min":5.4050000000000002,"15min":4.819},"max":{"1min":130.52500000000001,"5min":130.52500000000001,"15min":310.18299999999999},"last":11.693},{"interface":"front","average":{"1min":23.145,"5min":17.341000000000001,"15min":20.655000000000001},"min":{"1min":7.8479999999999999,"5min":6.4640000000000004,"15min":4.9290000000000003},"max":{"1min":113.23099999999999,"5min":113.23099999999999,"15min":263.452},"last":13.411}]},{"osd":2,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.693000000000001,"5min":19.597000000000001,"15min":21.677},"min":{"1min":6.149,"5min":6.149,"15min":6.149},"max":{"1min":70.923000000000002,"5min":142.77699999999999,"15min":250.30600000000001},"last":14.089},{"interface":"front","average":{"1min":16.704999999999998,"5min":18.449000000000002,"15min":20.291},"min":{"1min":8.2810000000000006,"5min":7.8360000000000003,"15min":6.4059999999999997},"max":{"1min":60.683,"5min":148.40100000000001,"15min":300.988},"last":9.4019999999999992}]},{"osd":3,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":17.314,"5min":22.145,"15min":23.927},"min":{"1min":7.9210000000000003,"5min":7.4370000000000003,"15min":6.9809999999999999},"max":{"1min":55.113999999999997,"5min":143.53899999999999,"15min":305.74900000000002},"last":12.702999999999999},{"interface":"front","average":{"1min":20.635999999999999,"5min":24.911999999999999,"15min":26.052},"min":{"1min":10.500999999999999,"5min":10.500999999999999,"15min":10.500999999999999},"max":{"1min":71.872,"5min":144.41800000000001,"15min":326.85899999999998},"last":17.547999999999998}]},{"osd":4,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":15.747999999999999,"5min":21.678000000000001,"15min":24.283000000000001},"min":{"1min":7.5999999999999996,"5min":6.5899999999999999,"15min":6.5899999999999999},"max":{"1min":68.769999999999996,"5min":136.691,"15min":320.23099999999999},"last":7.3150000000000004},{"interface":"front","average":{"1min":17.957000000000001,"5min":24.734000000000002,"15min":23.757999999999999},"min":{"1min":10.146000000000001,"5min":8.7840000000000007,"15min":6.6929999999999996},"max":{"1min":39.585999999999999,"5min":167.351,"15min":313.42500000000001},"last":15.571999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.943999999999999,"5min":25.558,"15min":30.716999999999999},"min":{"1min":10.445,"5min":9.1029999999999998,"15min":8.9280000000000008},"max":{"1min":35.082999999999998,"5min":201.87299999999999,"15min":320.84199999999998},"last":10.053000000000001},{"interface":"front","average":{"1min":18.722000000000001,"5min":25.786999999999999,"15min":30.187999999999999},"min":{"1min":11.132,"5min":8.3030000000000008,"15min":6.3630000000000004},"max":{"1min":32.395000000000003,"5min":192.81800000000001,"15min":365.91899999999998},"last":18.367999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":13.975,"5min":19.853000000000002,"15min":21.242999999999999},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.484},"max":{"1min":24.908999999999999,"5min":203.636,"15min":286.029},"last":12.359999999999999},{"interface":"front","average":{"1min":17.344999999999999,"5min":24.433,"15min":28.132999999999999},"min":{"1min":8.4990000000000006,"5min":8.4990000000000006,"15min":8.4990000000000006},"max":{"1min":31.541,"5min":204.45500000000001,"15min":300.18200000000002},"last":14.753}]}]},{"osd":4,"up_from":13,"seq":55834575115,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59932,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311908,"statfs":{"total":96636764160,"available":96575393792,"internally_reserved":0,"allocated":6705152,"data_stored":3683084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":10.369999999999999,"5min":13.005000000000001,"15min":17.228999999999999},"min":{"1min":4.5499999999999998,"5min":4.5499999999999998,"15min":4.3230000000000004},"max":{"1min":22.457999999999998,"5min":98.765000000000001,"15min":301.959},"last":6.4450000000000003},{"interface":"front","average":{"1min":11.028,"5min":15.567,"15min":21.140999999999998},"min":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"max":{"1min":24.654,"5min":110.508,"15min":363.75099999999998},"last":7.165}]},{"osd":1,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":14.859999999999999,"5min":15.579000000000001,"15min":20.225000000000001},"min":{"1min":6.4900000000000002,"5min":6.4900000000000002,"15min":5.577},"max":{"1min":57.762,"5min":121.346,"15min":219.19300000000001},"last":12.329000000000001},{"interface":"front","average":{"1min":13.955,"5min":16.913,"15min":20.986000000000001},"min":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3209999999999997},"max":{"1min":38.786000000000001,"5min":111.107,"15min":183.60400000000001},"last":13.404999999999999}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.715999999999999,"5min":21.856000000000002,"15min":20.428999999999998},"min":{"1min":8.4719999999999995,"5min":7.8289999999999997,"15min":6.8710000000000004},"max":{"1min":20.859000000000002,"5min":176.452,"15min":176.452},"last":14.619},{"interface":"front","average":{"1min":15.119999999999999,"5min":22.337,"15min":21.960000000000001},"min":{"1min":8.2330000000000005,"5min":7.6020000000000003,"15min":7.5940000000000003},"max":{"1min":19.547000000000001,"5min":239.935,"15min":239.935},"last":9.7300000000000004}]},{"osd":3,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":9.5579999999999998,"5min":18.57,"15min":22.199000000000002},"min":{"1min":5.968,"5min":5.968,"15min":5.968},"max":{"1min":13.237,"5min":177.48699999999999,"15min":246.523},"last":19.023},{"interface":"front","average":{"1min":14.81,"5min":20.649000000000001,"15min":18.181999999999999},"min":{"1min":7.4720000000000004,"5min":6.274,"15min":5.4240000000000004},"max":{"1min":20.555,"5min":183.75399999999999,"15min":183.75399999999999},"last":21.131}]},{"osd":5,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.641,"5min":15.577,"15min":19.568999999999999},"min":{"1min":8.4770000000000003,"5min":6.327,"15min":6.327},"max":{"1min":20.013000000000002,"5min":123.998,"15min":148.21000000000001},"last":10.952999999999999},{"interface":"front","average":{"1min":18.177,"5min":22.536000000000001,"15min":23.007999999999999},"min":{"1min":13.364000000000001,"5min":9.1270000000000007,"15min":6.46},"max":{"1min":25.591000000000001,"5min":224.55500000000001,"15min":224.55500000000001},"last":12.935}]},{"osd":6,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":13.776999999999999,"5min":20.428000000000001,"15min":23.344000000000001},"min":{"1min":8.4740000000000002,"5min":8.2129999999999992,"15min":7.8079999999999998},"max":{"1min":22.021000000000001,"5min":158.36099999999999,"15min":268.392},"last":15.465999999999999},{"interface":"front","average":{"1min":17.134,"5min":25.366,"15min":26.675999999999998},"min":{"1min":13.542999999999999,"5min":10.300000000000001,"15min":9.9299999999999997},"max":{"1min":23.768000000000001,"5min":243.40700000000001,"15min":243.40700000000001},"last":16.286999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.846,"5min":24.731999999999999,"15min":26.576000000000001},"min":{"1min":10.978,"5min":9.9939999999999998,"15min":9.2319999999999993},"max":{"1min":25.713000000000001,"5min":225.375,"15min":261.75099999999998},"last":17.248000000000001},{"interface":"front","average":{"1min":13.435,"5min":19.010999999999999,"15min":19.09},"min":{"1min":9.1319999999999997,"5min":7.181,"15min":6.7110000000000003},"max":{"1min":20.039000000000001,"5min":167.53700000000001,"15min":178.39699999999999},"last":8.3309999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:55:47.630 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-07T07:55:47.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.727+0000 7fa061305640 10 monclient: get_monmap_and_config 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.727+0000 7fa061305640 10 monclient: build_initial_monmap 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.727+0000 7fa061305640 10 monclient: monmap: 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:47.728847+0000 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:47.728847+0000 2023-12-07T07:55:47.730 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:47.731 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:47.731 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.728+0000 7fa061305640 1 Processor -- start 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.728+0000 7fa061305640 1 -- start start 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.728+0000 7fa061305640 10 monclient: init 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa05c0a8910 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa05c0a9b70 0x7fa05c13f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient(hunting): picked mon.noname-c con 0x7fa05c0a9b70 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:47.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa05c141f90 0x7fa05c1423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient(hunting): picked mon.noname-a con 0x7fa05c141f90 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c0a8cf0 con 0x7fa05c0a8910 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c0aabd0 con 0x7fa05c141f90 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c0aaeb0 con 0x7fa05c0a9b70 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa061305640 10 monclient(hunting): authenticate will time out at 2670.107437s 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05b7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa05c141f90 0x7fa05c1423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05a7fc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa05c0a9b70 0x7fa05c13f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05b7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa05c141f90 0x7fa05c1423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38554/0 (socket says 172.21.15.97:38554) 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05b7fe640 1 -- 172.21.15.97:0/2514396893 learned_addr learned my addr 172.21.15.97:0/2514396893 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05b7fe640 10 monclient(hunting): get_auth_request con 0x7fa05c141f90 auth_method 0 2023-12-07T07:55:47.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.729+0000 7fa05b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05a7fc640 10 monclient(hunting): get_auth_request con 0x7fa05c0a9b70 auth_method 0 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.730+0000 7fa05a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 10 monclient(hunting): handle_auth_done global_id 16104 payload 274 2023-12-07T07:55:47.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 1 -- 172.21.15.97:0/2514396893 >> v2:172.21.15.154:3300/0 conn(0x7fa05c0a9b70 msgr2=0x7fa05c13f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:47.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 1 --2- 172.21.15.97:0/2514396893 >> v2:172.21.15.154:3300/0 conn(0x7fa05c0a9b70 0x7fa05c13f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 1 -- 172.21.15.97:0/2514396893 >> v1:172.21.15.154:6789/0 conn(0x7fa05c0a8910 legacy=0x7fa05c13ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:47.736 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 10 monclient: _finish_hunting 0 2023-12-07T07:55:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 1 monclient: found mon.noname-a 2023-12-07T07:55:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.732+0000 7fa05b7fe640 1 -- 172.21.15.97:0/2514396893 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c0a9720 con 0x7fa05c141f90 2023-12-07T07:55:47.737 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.733+0000 7fa05b7fe640 1 --2- 172.21.15.97:0/2514396893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c141f90 0x7fa05c1423f0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fa05c074590 tx=0x7fa05402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45abfc5da5dc1997 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 1 -- 172.21.15.97:0/2514396893 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05402ca80 con 0x7fa05c141f90 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: dump: 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.740 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: _finish_auth 0 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa061305640 5 monclient: authenticate success, global_id 16104 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa061305640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 1 -- 172.21.15.97:0/2514396893 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05402cbe0 con 0x7fa05c141f90 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa061305640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa061305640 4 set_mon_vals no callback set 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 1 -- 172.21.15.97:0/2514396893 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0540343f0 con 0x7fa05c141f90 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:47.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.737+0000 7fa059ffb640 10 monclient: dump: 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.742 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 10 monclient: shutdown 2023-12-07T07:55:47.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:47.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 -- 172.21.15.97:0/2514396893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c141f90 msgr2=0x7fa05c1423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 --2- 172.21.15.97:0/2514396893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c141f90 0x7fa05c1423f0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fa05c074590 tx=0x7fa05402bee0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 -- 172.21.15.97:0/2514396893 shutdown_connections 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 --2- 172.21.15.97:0/2514396893 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c141f90 0x7fa05c1423f0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 --2- 172.21.15.97:0/2514396893 >> v2:172.21.15.154:3300/0 conn(0x7fa05c0a9b70 0x7fa05c13f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 -- 172.21.15.97:0/2514396893 >> 172.21.15.97:0/2514396893 conn(0x7fa05c0b7780 msgr2=0x7fa05c0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 -- 172.21.15.97:0/2514396893 shutdown_connections 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 1 -- 172.21.15.97:0/2514396893 wait complete. 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 10 monclient: build_initial_monmap 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.738+0000 7fa061305640 10 monclient: monmap: 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.746 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.739+0000 7fa061305640 1 librados: starting msgr at 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.739+0000 7fa061305640 1 librados: starting objecter 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.739+0000 7fa061305640 1 Processor -- start 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.739+0000 7fa061305640 1 -- start start 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.739+0000 7fa061305640 1 librados: setting wanted keys 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.739+0000 7fa061305640 1 librados: calling monclient init 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.739+0000 7fa061305640 10 monclient: init 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa05c0a8910 0x7fa05c0b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): picked mon.noname-b con 0x7fa05c0a8910 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa05c141f90 0x7fa05c137a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): picked mon.noname-c con 0x7fa05c141f90 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c13b330 0x7fa05c13b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): picked mon.noname-a con 0x7fa05c13b330 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c138f70 con 0x7fa05c13b330 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c139250 con 0x7fa05c0a8910 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c137f90 con 0x7fa05c141f90 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa061305640 10 monclient(hunting): authenticate will time out at 2670.118385s 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05b7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c13b330 0x7fa05c13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05a7fc640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa05c141f90 0x7fa05c137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa05c0a8910 0x7fa05c0b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05b7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c13b330 0x7fa05c13b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38556/0 (socket says 172.21.15.97:38556) 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05b7fe640 1 -- 172.21.15.97:0/4091820210 learned_addr learned my addr 172.21.15.97:0/4091820210 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05b7fe640 10 monclient(hunting): get_auth_request con 0x7fa05c13b330 auth_method 0 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.740+0000 7fa05b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05affd640 10 monclient(hunting): get_auth_request con 0x7fa05c0a8910 auth_method 0 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05a7fc640 10 monclient(hunting): get_auth_request con 0x7fa05c141f90 auth_method 0 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:47.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:47.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.741+0000 7fa05a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 10 monclient(hunting): handle_auth_done global_id 20173 payload 824 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 1 -- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa05c141f90 msgr2=0x7fa05c137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa05c141f90 0x7fa05c137a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 1 -- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c13b330 msgr2=0x7fa05c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c13b330 0x7fa05c13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 10 monclient: _finish_hunting 0 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 1 monclient: found mon.noname-b 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:47.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.744+0000 7fa05affd640 1 -- 172.21.15.97:0/4091820210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa054009b30 con 0x7fa05c0a8910 2023-12-07T07:55:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.746+0000 7fa05affd640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa05c0a8910 0x7fa05c0b70a0 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00c410 tx=0x7fa04c00c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.746+0000 7fa0437fe640 10 client.?.objecter ms_handle_connect 0x7fa05c0a8910 2023-12-07T07:55:47.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.746+0000 7fa0437fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 1 -- 172.21.15.97:0/4091820210 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0035f0 con 0x7fa05c0a8910 2023-12-07T07:55:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: dump: 2023-12-07T07:55:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.753 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: _finish_auth 0 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: _check_auth_tickets 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 1 -- 172.21.15.97:0/4091820210 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c003770 con 0x7fa05c0a8910 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 5 monclient: authenticate success, global_id 20173 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 10 monclient: _renew_subs 2023-12-07T07:55:47.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c136ea0 con 0x7fa05c0a8910 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 1 -- 172.21.15.97:0/4091820210 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c003b20 con 0x7fa05c0a8910 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0437fe640 10 monclient: dump: 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:47.755 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 10 client.20173.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 10 monclient: _renew_subs 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa0597fa640 4 set_mon_vals no callback set 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c1396e0 con 0x7fa05c0a8910 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.750+0000 7fa061305640 1 librados: init done 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.751+0000 7fa061305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.751+0000 7fa061305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.751+0000 7fa061305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:47.756 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.751+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c075b60 con 0x7fa05c0a8910 2023-12-07T07:55:47.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.771+0000 7fa0437fe640 1 -- 172.21.15.97:0/4091820210 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa04c003c80 con 0x7fa05c0a8910 2023-12-07T07:55:47.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.771+0000 7fa0437fe640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa04806c820 0x7fa04806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:47.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.772+0000 7fa05a7fc640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa04806c820 0x7fa04806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:47.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.772+0000 7fa05a7fc640 10 monclient: get_auth_request con 0x7fa04806c820 auth_method 0 2023-12-07T07:55:47.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.772+0000 7fa05a7fc640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa04806c820 0x7fa04806ece0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa05c138500 tx=0x7fa050009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:47.775 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.772+0000 7fa0437fe640 10 client.20173.objecter ms_handle_connect 0x7fa04806c820 2023-12-07T07:55:47.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.775+0000 7fa0437fe640 1 -- 172.21.15.97:0/4091820210 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fa04c057e70 con 0x7fa05c0a8910 2023-12-07T07:55:47.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.775+0000 7fa0437fe640 10 client.20173.objecter ms_dispatch 0x7fa05c000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:47.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.775+0000 7fa0437fe640 3 client.20173.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:47.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.775+0000 7fa0437fe640 3 client.20173.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:47.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:47.775+0000 7fa0437fe640 20 client.20173.objecter dump_active .. 0 homeless 2023-12-07T07:55:48.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.011+0000 7fa0437fe640 1 -- 172.21.15.97:0/4091820210 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa04c0541b0 con 0x7fa05c0a8910 2023-12-07T07:55:48.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.012+0000 7fa0437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:48.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.012+0000 7fa0437fe640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:48.193 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.191+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa05c139390 con 0x7fa04806c820 2023-12-07T07:55:48.198 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.196+0000 7fa0437fe640 1 -- 172.21.15.97:0/4091820210 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151872 (secure 0 0 0) 0x7fa050011a30 con 0x7fa04806c820 2023-12-07T07:55:48.199 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:55:48.200 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 10 librados: watch_flush enter 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 10 librados: watch_flush exit 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 20 client.20173.objecter shutdown clearing up homeless session... 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 10 client.20173.objecter successfully canceled tick 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa04806c820 msgr2=0x7fa04806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa04806c820 0x7fa04806ece0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa05c138500 tx=0x7fa050009190 comp rx=0 tx=0).stop 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 10 monclient: shutdown 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa05c0a8910 msgr2=0x7fa05c0b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa05c0a8910 0x7fa05c0b70a0 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00c410 tx=0x7fa04c00c810 comp rx=0 tx=0).stop 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 shutdown_connections 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fa04806c820 0x7fa04806ece0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.202 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa05c13b330 0x7fa05c13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa05c141f90 0x7fa05c137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 --2- 172.21.15.97:0/4091820210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa05c0a8910 0x7fa05c0b70a0 unknown :-1 s=CLOSED pgs=4853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 >> 172.21.15.97:0/4091820210 conn(0x7fa05c0b7780 msgr2=0x7fa05c13f180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:48.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 shutdown_connections 2023-12-07T07:55:48.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 -- 172.21.15.97:0/4091820210 wait complete. 2023-12-07T07:55:48.203 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.198+0000 7fa061305640 1 librados: shutdown 2023-12-07T07:55:48.211 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2023-12-07T07:55:47.963470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542992,"kb_used_data":53380,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754431728,"statfs":{"total":773094113280,"available":772538089472,"internally_reserved":0,"allocated":54661120,"data_stored":30274245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":501206576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":360,"apply_latency_ms":360,"commit_latency_ns":360000000,"apply_latency_ns":360000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.403143"},"pg_stats":[{"pgid":"11.17","version":"0'0","reported_seq":861,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.459982+0000","last_change":"2023-12-07T07:34:49.234367+0000","last_active":"2023-12-07T07:55:40.459982+0000","last_peered":"2023-12-07T07:55:40.459982+0000","last_clean":"2023-12-07T07:55:40.459982+0000","last_became_active":"2023-12-07T07:34:49.221580+0000","last_became_peered":"2023-12-07T07:34:49.221580+0000","last_unstale":"2023-12-07T07:55:40.459982+0000","last_undegraded":"2023-12-07T07:55:40.459982+0000","last_fullsized":"2023-12-07T07:55:40.459982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:19:37.266783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.138320+0000","last_change":"2023-12-07T07:34:47.569420+0000","last_active":"2023-12-07T07:55:43.138320+0000","last_peered":"2023-12-07T07:55:43.138320+0000","last_clean":"2023-12-07T07:55:43.138320+0000","last_became_active":"2023-12-07T07:34:47.563950+0000","last_became_peered":"2023-12-07T07:34:47.563950+0000","last_unstale":"2023-12-07T07:55:43.138320+0000","last_undegraded":"2023-12-07T07:55:43.138320+0000","last_fullsized":"2023-12-07T07:55:43.138320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:43.199927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":878,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.524025+0000","last_change":"2023-12-07T07:34:47.040719+0000","last_active":"2023-12-07T07:55:46.524025+0000","last_peered":"2023-12-07T07:55:46.524025+0000","last_clean":"2023-12-07T07:55:46.524025+0000","last_became_active":"2023-12-07T07:34:47.035305+0000","last_became_peered":"2023-12-07T07:34:47.035305+0000","last_unstale":"2023-12-07T07:55:46.524025+0000","last_undegraded":"2023-12-07T07:55:46.524025+0000","last_fullsized":"2023-12-07T07:55:46.524025+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:31:49.037782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":879,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.090489+0000","last_change":"2023-12-07T07:34:55.054864+0000","last_active":"2023-12-07T07:55:44.090489+0000","last_peered":"2023-12-07T07:55:44.090489+0000","last_clean":"2023-12-07T07:55:44.090489+0000","last_became_active":"2023-12-07T07:34:55.049584+0000","last_became_peered":"2023-12-07T07:34:55.049584+0000","last_unstale":"2023-12-07T07:55:44.090489+0000","last_undegraded":"2023-12-07T07:55:44.090489+0000","last_fullsized":"2023-12-07T07:55:44.090489+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:33:36.960338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.085712+0000","last_change":"2023-12-07T07:35:04.485603+0000","last_active":"2023-12-07T07:55:43.085712+0000","last_peered":"2023-12-07T07:55:43.085712+0000","last_clean":"2023-12-07T07:55:43.085712+0000","last_became_active":"2023-12-07T07:35:04.480195+0000","last_became_peered":"2023-12-07T07:35:04.480195+0000","last_unstale":"2023-12-07T07:55:43.085712+0000","last_undegraded":"2023-12-07T07:55:43.085712+0000","last_fullsized":"2023-12-07T07:55:43.085712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:46:21.908088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.299869+0000","last_change":"2023-12-07T07:34:51.744201+0000","last_active":"2023-12-07T07:55:44.299869+0000","last_peered":"2023-12-07T07:55:44.299869+0000","last_clean":"2023-12-07T07:55:44.299869+0000","last_became_active":"2023-12-07T07:34:51.735923+0000","last_became_peered":"2023-12-07T07:34:51.735923+0000","last_unstale":"2023-12-07T07:55:44.299869+0000","last_undegraded":"2023-12-07T07:55:44.299869+0000","last_fullsized":"2023-12-07T07:55:44.299869+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:23.046164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":891,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.062176+0000","last_change":"2023-12-07T07:34:47.067218+0000","last_active":"2023-12-07T07:55:46.062176+0000","last_peered":"2023-12-07T07:55:46.062176+0000","last_clean":"2023-12-07T07:55:46.062176+0000","last_became_active":"2023-12-07T07:34:47.052848+0000","last_became_peered":"2023-12-07T07:34:47.052848+0000","last_unstale":"2023-12-07T07:55:46.062176+0000","last_undegraded":"2023-12-07T07:55:46.062176+0000","last_fullsized":"2023-12-07T07:55:46.062176+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:34:00.206441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":879,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.569736+0000","last_change":"2023-12-07T07:34:51.124246+0000","last_active":"2023-12-07T07:55:46.569736+0000","last_peered":"2023-12-07T07:55:46.569736+0000","last_clean":"2023-12-07T07:55:46.569736+0000","last_became_active":"2023-12-07T07:34:51.115413+0000","last_became_peered":"2023-12-07T07:34:51.115413+0000","last_unstale":"2023-12-07T07:55:46.569736+0000","last_undegraded":"2023-12-07T07:55:46.569736+0000","last_fullsized":"2023-12-07T07:55:46.569736+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:15:19.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":892,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.963691+0000","last_change":"2023-12-07T07:34:47.170254+0000","last_active":"2023-12-07T07:55:45.963691+0000","last_peered":"2023-12-07T07:55:45.963691+0000","last_clean":"2023-12-07T07:55:45.963691+0000","last_became_active":"2023-12-07T07:34:47.149617+0000","last_became_peered":"2023-12-07T07:34:47.149617+0000","last_unstale":"2023-12-07T07:55:45.963691+0000","last_undegraded":"2023-12-07T07:55:45.963691+0000","last_fullsized":"2023-12-07T07:55:45.963691+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:02:51.144651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.081083+0000","last_change":"2023-12-07T07:34:53.278594+0000","last_active":"2023-12-07T07:55:44.081083+0000","last_peered":"2023-12-07T07:55:44.081083+0000","last_clean":"2023-12-07T07:55:44.081083+0000","last_became_active":"2023-12-07T07:34:53.269724+0000","last_became_peered":"2023-12-07T07:34:53.269724+0000","last_unstale":"2023-12-07T07:55:44.081083+0000","last_undegraded":"2023-12-07T07:55:44.081083+0000","last_fullsized":"2023-12-07T07:55:44.081083+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:30:32.694489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":864,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.104600+0000","last_change":"2023-12-07T07:34:56.759465+0000","last_active":"2023-12-07T07:55:43.104600+0000","last_peered":"2023-12-07T07:55:43.104600+0000","last_clean":"2023-12-07T07:55:43.104600+0000","last_became_active":"2023-12-07T07:34:56.746142+0000","last_became_peered":"2023-12-07T07:34:56.746142+0000","last_unstale":"2023-12-07T07:55:43.104600+0000","last_undegraded":"2023-12-07T07:55:43.104600+0000","last_fullsized":"2023-12-07T07:55:43.104600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T16:37:43.750416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.314285+0000","last_change":"2023-12-07T07:34:49.363608+0000","last_active":"2023-12-07T07:55:44.314285+0000","last_peered":"2023-12-07T07:55:44.314285+0000","last_clean":"2023-12-07T07:55:44.314285+0000","last_became_active":"2023-12-07T07:34:49.356256+0000","last_became_peered":"2023-12-07T07:34:49.356256+0000","last_unstale":"2023-12-07T07:55:44.314285+0000","last_undegraded":"2023-12-07T07:55:44.314285+0000","last_fullsized":"2023-12-07T07:55:44.314285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:27:32.615958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.083100+0000","last_change":"2023-12-07T07:33:42.670811+0000","last_active":"2023-12-07T07:55:44.083100+0000","last_peered":"2023-12-07T07:55:44.083100+0000","last_clean":"2023-12-07T07:55:44.083100+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:55:44.083100+0000","last_undegraded":"2023-12-07T07:55:44.083100+0000","last_fullsized":"2023-12-07T07:55:44.083100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:33:38.367199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":862,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.451617+0000","last_change":"2023-12-07T07:34:48.180457+0000","last_active":"2023-12-07T07:55:40.451617+0000","last_peered":"2023-12-07T07:55:40.451617+0000","last_clean":"2023-12-07T07:55:40.451617+0000","last_became_active":"2023-12-07T07:34:48.171478+0000","last_became_peered":"2023-12-07T07:34:48.171478+0000","last_unstale":"2023-12-07T07:55:40.451617+0000","last_undegraded":"2023-12-07T07:55:40.451617+0000","last_fullsized":"2023-12-07T07:55:40.451617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:42.397424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":878,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.913529+0000","last_change":"2023-12-07T07:33:43.216945+0000","last_active":"2023-12-07T07:55:45.913529+0000","last_peered":"2023-12-07T07:55:45.913529+0000","last_clean":"2023-12-07T07:55:45.913529+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:55:45.913529+0000","last_undegraded":"2023-12-07T07:55:45.913529+0000","last_fullsized":"2023-12-07T07:55:45.913529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:54.221713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":862,"reported_epoch":914,"state":"active+clean","last_fresh":"2023-12-07T07:55:40.458396+0000","last_change":"2023-12-07T07:34:49.998279+0000","last_active":"2023-12-07T07:55:40.458396+0000","last_peered":"2023-12-07T07:55:40.458396+0000","last_clean":"2023-12-07T07:55:40.458396+0000","last_became_active":"2023-12-07T07:34:49.980936+0000","last_became_peered":"2023-12-07T07:34:49.980936+0000","last_unstale":"2023-12-07T07:55:40.458396+0000","last_undegraded":"2023-12-07T07:55:40.458396+0000","last_fullsized":"2023-12-07T07:55:40.458396+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:00:32.835476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.059682+0000","last_change":"2023-12-07T07:33:42.426643+0000","last_active":"2023-12-07T07:55:44.059682+0000","last_peered":"2023-12-07T07:55:44.059682+0000","last_clean":"2023-12-07T07:55:44.059682+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:55:44.059682+0000","last_undegraded":"2023-12-07T07:55:44.059682+0000","last_fullsized":"2023-12-07T07:55:44.059682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:11:45.155210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":891,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.965586+0000","last_change":"2023-12-07T07:34:51.761228+0000","last_active":"2023-12-07T07:55:45.965586+0000","last_peered":"2023-12-07T07:55:45.965586+0000","last_clean":"2023-12-07T07:55:45.965586+0000","last_became_active":"2023-12-07T07:34:51.754214+0000","last_became_peered":"2023-12-07T07:34:51.754214+0000","last_unstale":"2023-12-07T07:55:45.965586+0000","last_undegraded":"2023-12-07T07:55:45.965586+0000","last_fullsized":"2023-12-07T07:55:45.965586+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:05:43.242940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.522315+0000","last_change":"2023-12-07T07:33:42.656077+0000","last_active":"2023-12-07T07:55:46.522315+0000","last_peered":"2023-12-07T07:55:46.522315+0000","last_clean":"2023-12-07T07:55:46.522315+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:55:46.522315+0000","last_undegraded":"2023-12-07T07:55:46.522315+0000","last_fullsized":"2023-12-07T07:55:46.522315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:05:05.686340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":891,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.961806+0000","last_change":"2023-12-07T07:35:04.576294+0000","last_active":"2023-12-07T07:55:45.961806+0000","last_peered":"2023-12-07T07:55:45.961806+0000","last_clean":"2023-12-07T07:55:45.961806+0000","last_became_active":"2023-12-07T07:35:04.566145+0000","last_became_peered":"2023-12-07T07:35:04.566145+0000","last_unstale":"2023-12-07T07:55:45.961806+0000","last_undegraded":"2023-12-07T07:55:45.961806+0000","last_fullsized":"2023-12-07T07:55:45.961806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:29.588414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.057960+0000","last_change":"2023-12-07T07:33:42.617026+0000","last_active":"2023-12-07T07:55:44.057960+0000","last_peered":"2023-12-07T07:55:44.057960+0000","last_clean":"2023-12-07T07:55:44.057960+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:55:44.057960+0000","last_undegraded":"2023-12-07T07:55:44.057960+0000","last_fullsized":"2023-12-07T07:55:44.057960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:13:58.336948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":872,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.372731+0000","last_change":"2023-12-07T07:34:47.445712+0000","last_active":"2023-12-07T07:55:46.372731+0000","last_peered":"2023-12-07T07:55:46.372731+0000","last_clean":"2023-12-07T07:55:46.372731+0000","last_became_active":"2023-12-07T07:34:47.437980+0000","last_became_peered":"2023-12-07T07:34:47.437980+0000","last_unstale":"2023-12-07T07:55:46.372731+0000","last_undegraded":"2023-12-07T07:55:46.372731+0000","last_fullsized":"2023-12-07T07:55:46.372731+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:52:30.429862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":896,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.956192+0000","last_change":"2023-12-07T07:33:42.269533+0000","last_active":"2023-12-07T07:55:45.956192+0000","last_peered":"2023-12-07T07:55:45.956192+0000","last_clean":"2023-12-07T07:55:45.956192+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:55:45.956192+0000","last_undegraded":"2023-12-07T07:55:45.956192+0000","last_fullsized":"2023-12-07T07:55:45.956192+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:12:50.238470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":872,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.358163+0000","last_change":"2023-12-07T07:35:04.029478+0000","last_active":"2023-12-07T07:55:46.358163+0000","last_peered":"2023-12-07T07:55:46.358163+0000","last_clean":"2023-12-07T07:55:46.358163+0000","last_became_active":"2023-12-07T07:35:04.015545+0000","last_became_peered":"2023-12-07T07:35:04.015545+0000","last_unstale":"2023-12-07T07:55:46.358163+0000","last_undegraded":"2023-12-07T07:55:46.358163+0000","last_fullsized":"2023-12-07T07:55:46.358163+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:16:10.231382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.093168+0000","last_change":"2023-12-07T07:33:42.349265+0000","last_active":"2023-12-07T07:55:44.093168+0000","last_peered":"2023-12-07T07:55:44.093168+0000","last_clean":"2023-12-07T07:55:44.093168+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:55:44.093168+0000","last_undegraded":"2023-12-07T07:55:44.093168+0000","last_fullsized":"2023-12-07T07:55:44.093168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:03:37.867596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":876,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.907529+0000","last_change":"2023-12-07T07:34:52.478105+0000","last_active":"2023-12-07T07:55:45.907529+0000","last_peered":"2023-12-07T07:55:45.907529+0000","last_clean":"2023-12-07T07:55:45.907529+0000","last_became_active":"2023-12-07T07:34:52.469358+0000","last_became_peered":"2023-12-07T07:34:52.469358+0000","last_unstale":"2023-12-07T07:55:45.907529+0000","last_undegraded":"2023-12-07T07:55:45.907529+0000","last_fullsized":"2023-12-07T07:55:45.907529+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:11:55.093716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":885,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.052648+0000","last_change":"2023-12-07T07:33:42.649680+0000","last_active":"2023-12-07T07:55:44.052648+0000","last_peered":"2023-12-07T07:55:44.052648+0000","last_clean":"2023-12-07T07:55:44.052648+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:55:44.052648+0000","last_undegraded":"2023-12-07T07:55:44.052648+0000","last_fullsized":"2023-12-07T07:55:44.052648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:13:55.895630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":976,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.077885+0000","last_change":"2023-12-07T07:33:36.384672+0000","last_active":"2023-12-07T07:55:43.077885+0000","last_peered":"2023-12-07T07:55:43.077885+0000","last_clean":"2023-12-07T07:55:43.077885+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:55:43.077885+0000","last_undegraded":"2023-12-07T07:55:43.077885+0000","last_fullsized":"2023-12-07T07:55:43.077885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_clean_scrub_stamp":"2023-12-07T07:33:30.679434+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:54:14.180629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.519129+0000","last_change":"2023-12-07T07:34:47.016526+0000","last_active":"2023-12-07T07:55:46.519129+0000","last_peered":"2023-12-07T07:55:46.519129+0000","last_clean":"2023-12-07T07:55:46.519129+0000","last_became_active":"2023-12-07T07:34:47.010508+0000","last_became_peered":"2023-12-07T07:34:47.010508+0000","last_unstale":"2023-12-07T07:55:46.519129+0000","last_undegraded":"2023-12-07T07:55:46.519129+0000","last_fullsized":"2023-12-07T07:55:46.519129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:26:28.206752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":873,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.360326+0000","last_change":"2023-12-07T07:35:02.508081+0000","last_active":"2023-12-07T07:55:46.360326+0000","last_peered":"2023-12-07T07:55:46.360326+0000","last_clean":"2023-12-07T07:55:46.360326+0000","last_became_active":"2023-12-07T07:35:02.498406+0000","last_became_peered":"2023-12-07T07:35:02.498406+0000","last_unstale":"2023-12-07T07:55:46.360326+0000","last_undegraded":"2023-12-07T07:55:46.360326+0000","last_fullsized":"2023-12-07T07:55:46.360326+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:26:16.927736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1","version":"38'7","reported_seq":878,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.494223+0000","last_change":"2023-12-07T07:34:48.573706+0000","last_active":"2023-12-07T07:55:41.494223+0000","last_peered":"2023-12-07T07:55:41.494223+0000","last_clean":"2023-12-07T07:55:41.494223+0000","last_became_active":"2023-12-07T07:34:48.562400+0000","last_became_peered":"2023-12-07T07:34:48.562400+0000","last_unstale":"2023-12-07T07:55:41.494223+0000","last_undegraded":"2023-12-07T07:55:41.494223+0000","last_fullsized":"2023-12-07T07:55:41.494223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:31:30.011782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.520737+0000","last_change":"2023-12-07T07:34:50.852474+0000","last_active":"2023-12-07T07:55:46.520737+0000","last_peered":"2023-12-07T07:55:46.520737+0000","last_clean":"2023-12-07T07:55:46.520737+0000","last_became_active":"2023-12-07T07:34:50.845529+0000","last_became_peered":"2023-12-07T07:34:50.845529+0000","last_unstale":"2023-12-07T07:55:46.520737+0000","last_undegraded":"2023-12-07T07:55:46.520737+0000","last_fullsized":"2023-12-07T07:55:46.520737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:04:08.453958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":881,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.049997+0000","last_change":"2023-12-07T07:34:44.988549+0000","last_active":"2023-12-07T07:55:44.049997+0000","last_peered":"2023-12-07T07:55:44.049997+0000","last_clean":"2023-12-07T07:55:44.049997+0000","last_became_active":"2023-12-07T07:34:44.899627+0000","last_became_peered":"2023-12-07T07:34:44.899627+0000","last_unstale":"2023-12-07T07:55:44.049997+0000","last_undegraded":"2023-12-07T07:55:44.049997+0000","last_fullsized":"2023-12-07T07:55:44.049997+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:34:53.809775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":865,"reported_epoch":915,"state":"active+clean","last_fresh":"2023-12-07T07:55:41.533937+0000","last_change":"2023-12-07T07:34:50.497421+0000","last_active":"2023-12-07T07:55:41.533937+0000","last_peered":"2023-12-07T07:55:41.533937+0000","last_clean":"2023-12-07T07:55:41.533937+0000","last_became_active":"2023-12-07T07:34:50.418281+0000","last_became_peered":"2023-12-07T07:34:50.418281+0000","last_unstale":"2023-12-07T07:55:41.533937+0000","last_undegraded":"2023-12-07T07:55:41.533937+0000","last_fullsized":"2023-12-07T07:55:41.533937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:26:13.156150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":866,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.106129+0000","last_change":"2023-12-07T07:34:58.289194+0000","last_active":"2023-12-07T07:55:43.106129+0000","last_peered":"2023-12-07T07:55:43.106129+0000","last_clean":"2023-12-07T07:55:43.106129+0000","last_became_active":"2023-12-07T07:34:58.263210+0000","last_became_peered":"2023-12-07T07:34:58.263210+0000","last_unstale":"2023-12-07T07:55:43.106129+0000","last_undegraded":"2023-12-07T07:55:43.106129+0000","last_fullsized":"2023-12-07T07:55:43.106129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:45:56.554479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.529472+0000","last_change":"2023-12-07T07:34:47.231473+0000","last_active":"2023-12-07T07:55:46.529472+0000","last_peered":"2023-12-07T07:55:46.529472+0000","last_clean":"2023-12-07T07:55:46.529472+0000","last_became_active":"2023-12-07T07:34:47.225369+0000","last_became_peered":"2023-12-07T07:34:47.225369+0000","last_unstale":"2023-12-07T07:55:46.529472+0000","last_undegraded":"2023-12-07T07:55:46.529472+0000","last_fullsized":"2023-12-07T07:55:46.529472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:53:23.919797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":865,"reported_epoch":916,"state":"active+clean","last_fresh":"2023-12-07T07:55:43.164122+0000","last_change":"2023-12-07T07:35:00.370860+0000","last_active":"2023-12-07T07:55:43.164122+0000","last_peered":"2023-12-07T07:55:43.164122+0000","last_clean":"2023-12-07T07:55:43.164122+0000","last_became_active":"2023-12-07T07:35:00.363892+0000","last_became_peered":"2023-12-07T07:35:00.363892+0000","last_unstale":"2023-12-07T07:55:43.164122+0000","last_undegraded":"2023-12-07T07:55:43.164122+0000","last_fullsized":"2023-12-07T07:55:43.164122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:52:59.781747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":871,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.355969+0000","last_change":"2023-12-07T07:34:45.545881+0000","last_active":"2023-12-07T07:55:46.355969+0000","last_peered":"2023-12-07T07:55:46.355969+0000","last_clean":"2023-12-07T07:55:46.355969+0000","last_became_active":"2023-12-07T07:34:45.533595+0000","last_became_peered":"2023-12-07T07:34:45.533595+0000","last_unstale":"2023-12-07T07:55:46.355969+0000","last_undegraded":"2023-12-07T07:55:46.355969+0000","last_fullsized":"2023-12-07T07:55:46.355969+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:06:30.943446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":875,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.905930+0000","last_change":"2023-12-07T07:34:54.924870+0000","last_active":"2023-12-07T07:55:45.905930+0000","last_peered":"2023-12-07T07:55:45.905930+0000","last_clean":"2023-12-07T07:55:45.905930+0000","last_became_active":"2023-12-07T07:34:54.918404+0000","last_became_peered":"2023-12-07T07:34:54.918404+0000","last_unstale":"2023-12-07T07:55:45.905930+0000","last_undegraded":"2023-12-07T07:55:45.905930+0000","last_fullsized":"2023-12-07T07:55:45.905930+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:07:29.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":879,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.056428+0000","last_change":"2023-12-07T07:34:45.422416+0000","last_active":"2023-12-07T07:55:44.056428+0000","last_peered":"2023-12-07T07:55:44.056428+0000","last_clean":"2023-12-07T07:55:44.056428+0000","last_became_active":"2023-12-07T07:34:45.408105+0000","last_became_peered":"2023-12-07T07:34:45.408105+0000","last_unstale":"2023-12-07T07:55:44.056428+0000","last_undegraded":"2023-12-07T07:55:44.056428+0000","last_fullsized":"2023-12-07T07:55:44.056428+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:54:33.469967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.309886+0000","last_change":"2023-12-07T07:34:49.403851+0000","last_active":"2023-12-07T07:55:44.309886+0000","last_peered":"2023-12-07T07:55:44.309886+0000","last_clean":"2023-12-07T07:55:44.309886+0000","last_became_active":"2023-12-07T07:34:49.378907+0000","last_became_peered":"2023-12-07T07:34:49.378907+0000","last_unstale":"2023-12-07T07:55:44.309886+0000","last_undegraded":"2023-12-07T07:55:44.309886+0000","last_fullsized":"2023-12-07T07:55:44.309886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:27:11.061093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444299,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77976,"kb_used_data":6540,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293864,"statfs":{"total":96636764160,"available":96556916736,"internally_reserved":0,"allocated":6696960,"data_stored":3657210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":25.919,"15min":28.937999999999999},"min":{"1min":6.7430000000000003,"5min":5.7389999999999999,"15min":3.8220000000000001},"max":{"1min":57.213999999999999,"5min":240.48099999999999,"15min":963.553},"last":100.182},{"interface":"front","average":{"1min":21.895,"5min":32.722000000000001,"15min":26.271999999999998},"min":{"1min":8.7810000000000006,"5min":8.7810000000000006,"15min":4.7640000000000002},"max":{"1min":123.477,"5min":252.37899999999999,"15min":285.09399999999999},"last":23.638999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":9.9960000000000004,"5min":20.056000000000001,"15min":19.806999999999999},"min":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":5.5049999999999999},"max":{"1min":35.368000000000002,"5min":236.608,"15min":240.595},"last":7.4429999999999996},{"interface":"front","average":{"1min":15.894,"5min":28.577000000000002,"15min":25.738},"min":{"1min":9.3689999999999998,"5min":9.0139999999999993,"15min":6.3490000000000002},"max":{"1min":33.170000000000002,"5min":275.33300000000003,"15min":307.64100000000002},"last":8.0999999999999996}]},{"osd":2,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.742000000000001,"5min":29.213999999999999,"15min":23.890999999999998},"min":{"1min":10.449,"5min":9.9049999999999994,"15min":7.4749999999999996},"max":{"1min":56.265000000000001,"5min":277.43099999999998,"15min":277.43099999999998},"last":10.733000000000001},{"interface":"front","average":{"1min":12.913,"5min":22.611999999999998,"15min":24.943000000000001},"min":{"1min":6.3700000000000001,"5min":6.3700000000000001,"15min":6.3700000000000001},"max":{"1min":51.895000000000003,"5min":243.97800000000001,"15min":243.97800000000001},"last":8.7349999999999994}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.201000000000001,"5min":21.512,"15min":22.870000000000001},"min":{"1min":5.7670000000000003,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":37.640999999999998,"5min":195.59299999999999,"15min":314.84300000000002},"last":10.010999999999999},{"interface":"front","average":{"1min":18.228000000000002,"5min":27.143000000000001,"15min":26.853999999999999},"min":{"1min":9.6170000000000009,"5min":7.1500000000000004,"15min":7.1500000000000004},"max":{"1min":66.102999999999994,"5min":215.51900000000001,"15min":252.49799999999999},"last":14.25}]},{"osd":5,"last update":"Thu Dec 7 07:55:22 2023","interfaces":[{"interface":"back","average":{"1min":19.082999999999998,"5min":28.425999999999998,"15min":28.632999999999999},"min":{"1min":7.2640000000000002,"5min":7.2640000000000002,"15min":7.2640000000000002},"max":{"1min":76.856999999999999,"5min":241.82300000000001,"15min":241.82300000000001},"last":13.388},{"interface":"front","average":{"1min":20.713000000000001,"5min":30.940000000000001,"15min":30.433},"min":{"1min":9.3460000000000001,"5min":9.3460000000000001,"15min":8.6780000000000008},"max":{"1min":57.811,"5min":286.17200000000003,"15min":286.17200000000003},"last":22.283000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.530999999999999,"5min":21.175999999999998,"15min":26.318000000000001},"min":{"1min":7.0590000000000002,"5min":6.149,"15min":6.149},"max":{"1min":84.808000000000007,"5min":168.52600000000001,"15min":333.97000000000003},"last":9.2919999999999998},{"interface":"front","average":{"1min":19.992000000000001,"5min":28.367000000000001,"15min":28.050999999999998},"min":{"1min":10.449999999999999,"5min":10.449999999999999,"15min":9.0559999999999992},"max":{"1min":84.061999999999998,"5min":168.88499999999999,"15min":301.428},"last":24.472000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":24.667000000000002,"5min":28.010000000000002,"15min":28.553999999999998},"min":{"1min":12.593999999999999,"5min":8.9139999999999997,"15min":8.0549999999999997},"max":{"1min":83.186000000000007,"5min":186.81299999999999,"15min":247.31100000000001},"last":22.939},{"interface":"front","average":{"1min":19.212,"5min":20.692,"15min":24.864999999999998},"min":{"1min":8.0640000000000001,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":61.241999999999997,"5min":184.453,"15min":311.41000000000003},"last":11.443}]}]},{"osd":2,"up_from":17,"seq":73014444299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64108,"kb_used_data":6552,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307732,"statfs":{"total":96636764160,"available":96571117568,"internally_reserved":0,"allocated":6709248,"data_stored":3663823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":17.835999999999999,"5min":15.121,"15min":16.925000000000001},"min":{"1min":5.9960000000000004,"5min":5.9960000000000004,"15min":4.9409999999999998},"max":{"1min":111.95099999999999,"5min":149.566,"15min":339.37299999999999},"last":9.9649999999999999},{"interface":"front","average":{"1min":21.946999999999999,"5min":20.587,"15min":23.597999999999999},"min":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":4.6319999999999997},"max":{"1min":123.776,"5min":157.69300000000001,"15min":597.92399999999998},"last":11.215}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.486000000000001,"5min":18.428000000000001,"15min":21.236999999999998},"min":{"1min":7.4930000000000003,"5min":7.4930000000000003,"15min":4.5410000000000004},"max":{"1min":58.052999999999997,"5min":104.455,"15min":282.435},"last":10.6},{"interface":"front","average":{"1min":13.083,"5min":13.429,"15min":18.023},"min":{"1min":5.093,"5min":5.093,"15min":5.093},"max":{"1min":51.125,"5min":100.026,"15min":135.71000000000001},"last":8.7319999999999993}]},{"osd":3,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":21.981999999999999,"5min":23.928000000000001,"15min":20.510999999999999},"min":{"1min":10.015000000000001,"5min":8.1869999999999994,"15min":6.1699999999999999},"max":{"1min":47.886000000000003,"5min":160.94800000000001,"15min":160.94800000000001},"last":15.411},{"interface":"front","average":{"1min":16.581,"5min":18.431000000000001,"15min":21.231000000000002},"min":{"1min":6.5739999999999998,"5min":6.5339999999999998,"15min":6.5170000000000003},"max":{"1min":53.826000000000001,"5min":128.06100000000001,"15min":241.69},"last":8.1769999999999996}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":20.681999999999999,"5min":18.628,"15min":19.465},"min":{"1min":7.5700000000000003,"5min":7.0599999999999996,"15min":6.5570000000000004},"max":{"1min":69.706000000000003,"5min":130.738,"15min":229.14500000000001},"last":28.942},{"interface":"front","average":{"1min":24.361999999999998,"5min":22.751999999999999,"15min":21.934999999999999},"min":{"1min":10.942,"5min":10.942,"15min":5.9829999999999997},"max":{"1min":76.808000000000007,"5min":105.361,"15min":284.95100000000002},"last":25.314}]},{"osd":5,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.340999999999999,"5min":16.795999999999999,"15min":17.544},"min":{"1min":8.1159999999999997,"5min":6.0019999999999998,"15min":5.9489999999999998},"max":{"1min":43.698,"5min":144.864,"15min":247.37799999999999},"last":13.401},{"interface":"front","average":{"1min":18.547999999999998,"5min":21.937999999999999,"15min":21.414999999999999},"min":{"1min":10.209,"5min":6.7229999999999999,"15min":6.7229999999999999},"max":{"1min":57.433999999999997,"5min":143.517,"15min":218.82400000000001},"last":26.041}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":22.513000000000002,"5min":23.238,"15min":25.574000000000002},"min":{"1min":9.5549999999999997,"5min":7.7190000000000003,"15min":7.7190000000000003},"max":{"1min":64.820999999999998,"5min":133.203,"15min":288.38299999999998},"last":23.818999999999999},{"interface":"front","average":{"1min":21.914999999999999,"5min":25.452999999999999,"15min":29.690999999999999},"min":{"1min":12.843999999999999,"5min":10.220000000000001,"15min":8.25},"max":{"1min":55.670999999999999,"5min":182.745,"15min":291.435},"last":23.052}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.765999999999998,"5min":19.440000000000001,"15min":23.538},"min":{"1min":7.7519999999999998,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":45.223999999999997,"5min":162.23500000000001,"15min":302.30900000000003},"last":14.875},{"interface":"front","average":{"1min":22.626000000000001,"5min":25.323,"15min":24.372},"min":{"1min":10.784000000000001,"5min":9.3190000000000008,"15min":5.7859999999999996},"max":{"1min":66.337999999999994,"5min":136.339,"15min":279.00700000000001},"last":27.186}]}]},{"osd":1,"up_from":17,"seq":73014444300,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60328,"kb_used_data":6936,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311512,"statfs":{"total":96636764160,"available":96574988288,"internally_reserved":0,"allocated":7102464,"data_stored":4055055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":14.016,"5min":26.524000000000001,"15min":23.024000000000001},"min":{"1min":4.9160000000000004,"5min":4.3029999999999999,"15min":4.2080000000000002},"max":{"1min":93.808999999999997,"5min":348.17200000000003,"15min":410.08199999999999},"last":8.4000000000000004},{"interface":"front","average":{"1min":15.234999999999999,"5min":20.524999999999999,"15min":20.359000000000002},"min":{"1min":6.4189999999999996,"5min":5.5659999999999998,"15min":5.282},"max":{"1min":82.144999999999996,"5min":146.68700000000001,"15min":303.291},"last":21.623000000000001}]},{"osd":2,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":21.712,"15min":22.257999999999999},"min":{"1min":6.3479999999999999,"5min":4.4240000000000004,"15min":4.4240000000000004},"max":{"1min":23.940000000000001,"5min":192.226,"15min":336.255},"last":9.1639999999999997},{"interface":"front","average":{"1min":18.809999999999999,"5min":28.77,"15min":22.311},"min":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":5.2679999999999998},"max":{"1min":96.281000000000006,"5min":229.56100000000001,"15min":335.88499999999999},"last":21.283000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.492999999999999,"5min":28.306000000000001,"15min":25.484000000000002},"min":{"1min":9.2159999999999993,"5min":7.0449999999999999,"15min":6.3940000000000001},"max":{"1min":87.390000000000001,"5min":317.94999999999999,"15min":429.30399999999997},"last":24.088999999999999},{"interface":"front","average":{"1min":14.365,"5min":31.286000000000001,"15min":24.562000000000001},"min":{"1min":7.1550000000000002,"5min":6.3010000000000002,"15min":6.1589999999999998},"max":{"1min":46.652999999999999,"5min":302.00599999999997,"15min":382.79399999999998},"last":6.6289999999999996}]},{"osd":4,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.879999999999999,"5min":30.443999999999999,"15min":29.187000000000001},"min":{"1min":10.609,"5min":8.7759999999999998,"15min":5.9969999999999999},"max":{"1min":83.418000000000006,"5min":207.262,"15min":406.53899999999999},"last":24.789999999999999},{"interface":"front","average":{"1min":18.018999999999998,"5min":33.978999999999999,"15min":28.32},"min":{"1min":9.6039999999999992,"5min":7.4039999999999999,"15min":6.9279999999999999},"max":{"1min":96.866,"5min":417.81700000000001,"15min":417.81700000000001},"last":20.593}]},{"osd":5,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":23.068999999999999,"5min":27.93,"15min":28.184999999999999},"min":{"1min":11.724,"5min":10.911,"15min":7.3049999999999997},"max":{"1min":106.518,"5min":128.53899999999999,"15min":390.11000000000001},"last":25.827999999999999},{"interface":"front","average":{"1min":21.890000000000001,"5min":33.082000000000001,"15min":27.628},"min":{"1min":10.880000000000001,"5min":8.8249999999999993,"15min":7.0350000000000001},"max":{"1min":102.294,"5min":323.339,"15min":357.44799999999998},"last":22.367000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":22.527000000000001,"5min":29.898,"15min":31.405999999999999},"min":{"1min":11.798,"5min":10.257,"15min":7.9279999999999999},"max":{"1min":90.643000000000001,"5min":188.14400000000001,"15min":420.565},"last":28.507000000000001},{"interface":"front","average":{"1min":26.419,"5min":35.195999999999998,"15min":32.094999999999999},"min":{"1min":14.733000000000001,"5min":12.470000000000001,"15min":8.8680000000000003},"max":{"1min":94.477999999999994,"5min":264.226,"15min":340.71100000000001},"last":54.895000000000003}]},{"osd":7,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":17.436,"5min":24.896999999999998,"15min":27.373999999999999},"min":{"1min":8.2690000000000001,"5min":6.6500000000000004,"15min":6.6500000000000004},"max":{"1min":77.435000000000002,"5min":200.01900000000001,"15min":442.14100000000002},"last":10.077999999999999},{"interface":"front","average":{"1min":24.981000000000002,"5min":32.478999999999999,"15min":30.864000000000001},"min":{"1min":13.936999999999999,"5min":10.494999999999999,"15min":8.8699999999999992},"max":{"1min":98.965999999999994,"5min":267.07999999999998,"15min":533.87199999999996},"last":30.085999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60052,"kb_used_data":6656,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311788,"statfs":{"total":96636764160,"available":96575270912,"internally_reserved":0,"allocated":6815744,"data_stored":3789585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.449,"5min":20.334,"15min":21.199999999999999},"min":{"1min":7.2389999999999999,"5min":5.3540000000000001,"15min":5.3529999999999998},"max":{"1min":52.637,"5min":230.792,"15min":230.792},"last":8.4890000000000008},{"interface":"front","average":{"1min":17.687999999999999,"5min":22.419,"15min":27.146999999999998},"min":{"1min":8.0589999999999993,"5min":6.9210000000000003,"15min":6.9210000000000003},"max":{"1min":68.528000000000006,"5min":119.28,"15min":289.82400000000001},"last":10.679}]},{"osd":2,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":15.130000000000001,"5min":24.141999999999999,"15min":23.670999999999999},"min":{"1min":9.3160000000000007,"5min":8.8759999999999994,"15min":7.9080000000000004},"max":{"1min":26.367000000000001,"5min":201.53800000000001,"15min":201.53800000000001},"last":9.5359999999999996},{"interface":"front","average":{"1min":9.2829999999999995,"5min":17.088999999999999,"15min":16.57},"min":{"1min":5.8230000000000004,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":29.462,"5min":202.648,"15min":214.94},"last":8.7690000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":22.611000000000001,"15min":27.707000000000001},"min":{"1min":6.1550000000000002,"5min":5.3440000000000003,"15min":5.0899999999999999},"max":{"1min":240.876,"5min":283.43000000000001,"15min":932.75999999999999},"last":11.298},{"interface":"front","average":{"1min":20.475000000000001,"5min":26.277999999999999,"15min":27.437000000000001},"min":{"1min":11.481,"5min":8.0809999999999995,"15min":8.0809999999999995},"max":{"1min":78.986999999999995,"5min":256.267,"15min":300.13999999999999},"last":11.99}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":23.826000000000001,"5min":21.247,"15min":28.013000000000002},"min":{"1min":7.1529999999999996,"5min":6.7930000000000001,"15min":6.4649999999999999},"max":{"1min":153.63200000000001,"5min":153.63200000000001,"15min":1049.731},"last":226.23400000000001},{"interface":"front","average":{"1min":27.728000000000002,"5min":27.978999999999999,"15min":28.177},"min":{"1min":13.824,"5min":12.833,"15min":9.5399999999999991},"max":{"1min":108.58799999999999,"5min":158.40100000000001,"15min":283.43700000000001},"last":130.922}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":24.460999999999999,"5min":30.645,"15min":28.616},"min":{"1min":10.550000000000001,"5min":10.550000000000001,"15min":5.8300000000000001},"max":{"1min":105.548,"5min":248.34299999999999,"15min":255.328},"last":53.789000000000001},{"interface":"front","average":{"1min":20.219999999999999,"5min":23.016999999999999,"15min":30.934000000000001},"min":{"1min":6.8079999999999998,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":185.125,"5min":264.34899999999999,"15min":1217.117},"last":108.961}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.291,"5min":21.850000000000001,"15min":31.241},"min":{"1min":8.1869999999999994,"5min":7.8789999999999996,"15min":7.0350000000000001},"max":{"1min":28.216999999999999,"5min":154.68199999999999,"15min":1252.499},"last":13.973000000000001},{"interface":"front","average":{"1min":20.312000000000001,"5min":30.834,"15min":31.664999999999999},"min":{"1min":14.042,"5min":13.167999999999999,"15min":11.638},"max":{"1min":33.158000000000001,"5min":263.43200000000002,"15min":263.43200000000002},"last":13.231999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":22.698,"5min":32.343000000000004,"15min":32.625999999999998},"min":{"1min":15.154999999999999,"5min":13.768000000000001,"15min":13.212},"max":{"1min":37.085000000000001,"5min":140.50200000000001,"15min":279.38900000000001},"last":119.37},{"interface":"front","average":{"1min":14.250999999999999,"5min":24.257999999999999,"15min":23.280999999999999},"min":{"1min":7.8200000000000003,"5min":7.1459999999999999,"15min":6.2990000000000004},"max":{"1min":27.417000000000002,"5min":262.07299999999998,"15min":262.07299999999998},"last":215.11699999999999}]}]},{"osd":6,"up_from":14,"seq":60129542412,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59972,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311868,"statfs":{"total":96636764160,"available":96575352832,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":11.407,"5min":16.442,"15min":18.248999999999999},"min":{"1min":5.4500000000000002,"5min":5.4500000000000002,"15min":4.8579999999999997},"max":{"1min":30.831,"5min":227.49000000000001,"15min":227.49000000000001},"last":13.335000000000001},{"interface":"front","average":{"1min":19.172999999999998,"5min":22.824000000000002,"15min":24.23},"min":{"1min":6.343,"5min":6.343,"15min":5.9909999999999997},"max":{"1min":80.697000000000003,"5min":228.13399999999999,"15min":229.405},"last":15.199999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":14.680999999999999,"5min":21.905999999999999,"15min":22.215},"min":{"1min":8.3089999999999993,"5min":6.2599999999999998,"15min":5.9390000000000001},"max":{"1min":24.978000000000002,"5min":228.745,"15min":234.59399999999999},"last":10.336},{"interface":"front","average":{"1min":10.353,"5min":18.015999999999998,"15min":17.309000000000001},"min":{"1min":5.702,"5min":4.8090000000000002,"15min":4.6100000000000003},"max":{"1min":18.276,"5min":220.78800000000001,"15min":220.78800000000001},"last":7.0620000000000003}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":11.086,"5min":19.494,"15min":18.353000000000002},"min":{"1min":7.4279999999999999,"5min":5.8680000000000003,"15min":4.4210000000000003},"max":{"1min":30.722999999999999,"5min":156.13499999999999,"15min":215.05699999999999},"last":9.016},{"interface":"front","average":{"1min":16.837,"5min":26.483000000000001,"15min":25.437999999999999},"min":{"1min":10.422000000000001,"5min":7.9950000000000001,"15min":7.9950000000000001},"max":{"1min":26.327000000000002,"5min":232.45400000000001,"15min":232.45400000000001},"last":12.617000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":12.009,"5min":25.157,"15min":22.254999999999999},"min":{"1min":6.7590000000000003,"5min":6.024,"15min":6.024},"max":{"1min":26.135999999999999,"5min":248.97900000000001,"15min":248.97900000000001},"last":8.2859999999999996},{"interface":"front","average":{"1min":14.058,"5min":25.068000000000001,"15min":22.209},"min":{"1min":9.5719999999999992,"5min":6.0860000000000003,"15min":6.0860000000000003},"max":{"1min":27.628,"5min":236.92699999999999,"15min":236.92699999999999},"last":33.841999999999999}]},{"osd":4,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":19.373999999999999,"5min":27.763000000000002,"15min":23.887},"min":{"1min":6.3899999999999997,"5min":6.3899999999999997,"15min":6.3090000000000002},"max":{"1min":83.894000000000005,"5min":276.92500000000001,"15min":276.92500000000001},"last":38.558999999999997},{"interface":"front","average":{"1min":20.515999999999998,"5min":27.331,"15min":24.074000000000002},"min":{"1min":8.7449999999999992,"5min":7.258,"15min":6.5700000000000003},"max":{"1min":90.427000000000007,"5min":179.64099999999999,"15min":220.881},"last":32.463000000000001}]},{"osd":5,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":16.332000000000001,"5min":26.169,"15min":26.448},"min":{"1min":7.8570000000000002,"5min":7.8570000000000002,"15min":7.8570000000000002},"max":{"1min":34.262999999999998,"5min":237.53700000000001,"15min":237.53700000000001},"last":13.882999999999999},{"interface":"front","average":{"1min":18.193999999999999,"5min":27.972000000000001,"15min":25.896999999999998},"min":{"1min":11.1,"5min":9.1460000000000008,"15min":8.0099999999999998},"max":{"1min":29.687000000000001,"5min":239.673,"15min":239.673},"last":15.842000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":26.934999999999999,"5min":30.998999999999999,"15min":25.236999999999998},"min":{"1min":10.981999999999999,"5min":8.4890000000000008,"15min":7.3090000000000002},"max":{"1min":107.471,"5min":236.26499999999999,"15min":236.26499999999999},"last":33.204999999999998},{"interface":"front","average":{"1min":28.969999999999999,"5min":31.443999999999999,"15min":28.527999999999999},"min":{"1min":17.210999999999999,"5min":10.239000000000001,"15min":10.239000000000001},"max":{"1min":102.93600000000001,"5min":253.29300000000001,"15min":253.29300000000001},"last":31.664999999999999}]}]},{"osd":7,"up_from":13,"seq":55834575118,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78016,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293824,"statfs":{"total":96636764160,"available":96556875776,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":10.605,"5min":11.368,"15min":15.794},"min":{"1min":5.6929999999999996,"5min":4.3899999999999997,"15min":4.29},"max":{"1min":17.640999999999998,"5min":127.59,"15min":214.32599999999999},"last":6.3470000000000004},{"interface":"front","average":{"1min":10.164,"5min":10.188000000000001,"15min":15.063000000000001},"min":{"1min":4.3659999999999997,"5min":4.3659999999999997,"15min":4.3659999999999997},"max":{"1min":20.850000000000001,"5min":47.606000000000002,"15min":199.94499999999999},"last":10.141}]},{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":12.381,"5min":16.414000000000001,"15min":19.100000000000001},"min":{"1min":6.3719999999999999,"5min":5.8289999999999997,"15min":5.3849999999999998},"max":{"1min":25.638999999999999,"5min":157.16,"15min":198.512},"last":8.9879999999999995},{"interface":"front","average":{"1min":12.368,"5min":17.263999999999999,"15min":19.622},"min":{"1min":7.4349999999999996,"5min":5.7270000000000003,"15min":5.6929999999999996},"max":{"1min":18.352,"5min":167.78200000000001,"15min":167.78200000000001},"last":16.824000000000002}]},{"osd":2,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.574,"5min":18.367999999999999,"15min":18.265000000000001},"min":{"1min":7.3330000000000002,"5min":6.0049999999999999,"15min":5.9669999999999996},"max":{"1min":23.154,"5min":282.12099999999998,"15min":282.12099999999998},"last":15.484999999999999},{"interface":"front","average":{"1min":13.243,"5min":17.161000000000001,"15min":19.664000000000001},"min":{"1min":6.9939999999999998,"5min":6.9939999999999998,"15min":6.9939999999999998},"max":{"1min":20.405000000000001,"5min":154.376,"15min":216.28299999999999},"last":7.8179999999999996}]},{"osd":3,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":14.811999999999999,"5min":16.736999999999998,"15min":20.478999999999999},"min":{"1min":9.3559999999999999,"5min":6.4939999999999998,"15min":6.3410000000000002},"max":{"1min":21.123999999999999,"5min":84.263000000000005,"15min":159.934},"last":15.128},{"interface":"front","average":{"1min":15.291,"5min":18.553999999999998,"15min":20.609000000000002},"min":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":6.9409999999999998},"max":{"1min":28.382999999999999,"5min":147.85400000000001,"15min":157.43100000000001},"last":17.465}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":16.672000000000001,"5min":17.713999999999999,"15min":21.768000000000001},"min":{"1min":9.3829999999999991,"5min":7.2110000000000003,"15min":6.7000000000000002},"max":{"1min":33.926000000000002,"5min":94.227000000000004,"15min":204.71199999999999},"last":10.904999999999999},{"interface":"front","average":{"1min":18.524999999999999,"5min":18.350000000000001,"15min":21.297999999999998},"min":{"1min":9.1869999999999994,"5min":7.7839999999999998,"15min":7.1269999999999998},"max":{"1min":52.731000000000002,"5min":93.356999999999999,"15min":259.23200000000003},"last":19.602}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.744,"5min":21.847000000000001,"15min":25.094000000000001},"min":{"1min":10.131,"5min":8.9830000000000005,"15min":8.9830000000000005},"max":{"1min":92.340999999999994,"5min":189.49000000000001,"15min":239.32599999999999},"last":12.601000000000001},{"interface":"front","average":{"1min":15.037000000000001,"5min":18.690999999999999,"15min":23.567},"min":{"1min":8.3529999999999998,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":29.849,"5min":157.60400000000001,"15min":224.91900000000001},"last":18.765000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.780999999999999,"5min":20.637,"15min":24.367999999999999},"min":{"1min":12.939,"5min":9.9390000000000001,"15min":8.2750000000000004},"max":{"1min":49.506999999999998,"5min":66.572999999999993,"15min":192.76300000000001},"last":16.111999999999998},{"interface":"front","average":{"1min":20.835000000000001,"5min":23.234999999999999,"15min":29.07},"min":{"1min":13.638999999999999,"5min":13.432,"15min":10.176},"max":{"1min":34.186999999999998,"5min":92.688999999999993,"15min":216.917},"last":13.585000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575117,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82568,"kb_used_data":6968,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289272,"statfs":{"total":96636764160,"available":96552214528,"internally_reserved":0,"allocated":7135232,"data_stored":4075523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":11.474,"5min":12.582000000000001,"15min":22.064},"min":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":4.2409999999999997},"max":{"1min":27.518000000000001,"5min":130.309,"15min":1086.095},"last":7.7359999999999998},{"interface":"front","average":{"1min":11.550000000000001,"5min":16.629999999999999,"15min":20.43},"min":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"max":{"1min":38.128999999999998,"5min":141.636,"15min":403.21899999999999},"last":13.707000000000001}]},{"osd":1,"last update":"Thu Dec 7 07:55:36 2023","interfaces":[{"interface":"back","average":{"1min":20.469000000000001,"5min":15.369,"15min":17.274999999999999},"min":{"1min":6.234,"5min":5.4050000000000002,"15min":4.819},"max":{"1min":130.52500000000001,"5min":130.52500000000001,"15min":310.18299999999999},"last":12.348000000000001},{"interface":"front","average":{"1min":23.145,"5min":17.341000000000001,"15min":20.655000000000001},"min":{"1min":7.8479999999999999,"5min":6.4640000000000004,"15min":4.9290000000000003},"max":{"1min":113.23099999999999,"5min":113.23099999999999,"15min":263.452},"last":8.4329999999999998}]},{"osd":2,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.693000000000001,"5min":19.597000000000001,"15min":21.677},"min":{"1min":6.149,"5min":6.149,"15min":6.149},"max":{"1min":70.923000000000002,"5min":142.77699999999999,"15min":250.30600000000001},"last":15.315},{"interface":"front","average":{"1min":16.704999999999998,"5min":18.449000000000002,"15min":20.291},"min":{"1min":8.2810000000000006,"5min":7.8360000000000003,"15min":6.4059999999999997},"max":{"1min":60.683,"5min":148.40100000000001,"15min":300.988},"last":13.319000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":17.314,"5min":22.145,"15min":23.927},"min":{"1min":7.9210000000000003,"5min":7.4370000000000003,"15min":6.9809999999999999},"max":{"1min":55.113999999999997,"5min":143.53899999999999,"15min":305.74900000000002},"last":16.332000000000001},{"interface":"front","average":{"1min":20.635999999999999,"5min":24.911999999999999,"15min":26.052},"min":{"1min":10.500999999999999,"5min":10.500999999999999,"15min":10.500999999999999},"max":{"1min":71.872,"5min":144.41800000000001,"15min":326.85899999999998},"last":17.446000000000002}]},{"osd":4,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":15.747999999999999,"5min":21.678000000000001,"15min":24.283000000000001},"min":{"1min":7.5999999999999996,"5min":6.5899999999999999,"15min":6.5899999999999999},"max":{"1min":68.769999999999996,"5min":136.691,"15min":320.23099999999999},"last":20.451000000000001},{"interface":"front","average":{"1min":17.957000000000001,"5min":24.734000000000002,"15min":23.757999999999999},"min":{"1min":10.146000000000001,"5min":8.7840000000000007,"15min":6.6929999999999996},"max":{"1min":39.585999999999999,"5min":167.351,"15min":313.42500000000001},"last":18.483000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.943999999999999,"5min":25.558,"15min":30.716999999999999},"min":{"1min":10.445,"5min":9.1029999999999998,"15min":8.9280000000000008},"max":{"1min":35.082999999999998,"5min":201.87299999999999,"15min":320.84199999999998},"last":22.712},{"interface":"front","average":{"1min":18.722000000000001,"5min":25.786999999999999,"15min":30.187999999999999},"min":{"1min":11.132,"5min":8.3030000000000008,"15min":6.3630000000000004},"max":{"1min":32.395000000000003,"5min":192.81800000000001,"15min":365.91899999999998},"last":19.417999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":13.975,"5min":19.853000000000002,"15min":21.242999999999999},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.484},"max":{"1min":24.908999999999999,"5min":203.636,"15min":286.029},"last":21.891999999999999},{"interface":"front","average":{"1min":17.344999999999999,"5min":24.433,"15min":28.132999999999999},"min":{"1min":8.4990000000000006,"5min":8.4990000000000006,"15min":8.4990000000000006},"max":{"1min":31.541,"5min":204.45500000000001,"15min":300.18200000000002},"last":23.451000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575116,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59972,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311868,"statfs":{"total":96636764160,"available":96575352832,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":10.369999999999999,"5min":13.005000000000001,"15min":17.228999999999999},"min":{"1min":4.5499999999999998,"5min":4.5499999999999998,"15min":4.3230000000000004},"max":{"1min":22.457999999999998,"5min":98.765000000000001,"15min":301.959},"last":10.234},{"interface":"front","average":{"1min":11.028,"5min":15.567,"15min":21.140999999999998},"min":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"max":{"1min":24.654,"5min":110.508,"15min":363.75099999999998},"last":16.82}]},{"osd":1,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":14.859999999999999,"5min":15.579000000000001,"15min":20.225000000000001},"min":{"1min":6.4900000000000002,"5min":6.4900000000000002,"15min":5.577},"max":{"1min":57.762,"5min":121.346,"15min":219.19300000000001},"last":11.582000000000001},{"interface":"front","average":{"1min":13.955,"5min":16.913,"15min":20.986000000000001},"min":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3209999999999997},"max":{"1min":38.786000000000001,"5min":111.107,"15min":183.60400000000001},"last":8.9700000000000006}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.715999999999999,"5min":21.856000000000002,"15min":20.428999999999998},"min":{"1min":8.4719999999999995,"5min":7.8289999999999997,"15min":6.8710000000000004},"max":{"1min":20.859000000000002,"5min":176.452,"15min":176.452},"last":14.077999999999999},{"interface":"front","average":{"1min":15.119999999999999,"5min":22.337,"15min":21.960000000000001},"min":{"1min":8.2330000000000005,"5min":7.6020000000000003,"15min":7.5940000000000003},"max":{"1min":19.547000000000001,"5min":239.935,"15min":239.935},"last":12.212999999999999}]},{"osd":3,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":9.5579999999999998,"5min":18.57,"15min":22.199000000000002},"min":{"1min":5.968,"5min":5.968,"15min":5.968},"max":{"1min":13.237,"5min":177.48699999999999,"15min":246.523},"last":10.9},{"interface":"front","average":{"1min":14.81,"5min":20.649000000000001,"15min":18.181999999999999},"min":{"1min":7.4720000000000004,"5min":6.274,"15min":5.4240000000000004},"max":{"1min":20.555,"5min":183.75399999999999,"15min":183.75399999999999},"last":8.1950000000000003}]},{"osd":5,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.641,"5min":15.577,"15min":19.568999999999999},"min":{"1min":8.4770000000000003,"5min":6.327,"15min":6.327},"max":{"1min":20.013000000000002,"5min":123.998,"15min":148.21000000000001},"last":6.8380000000000001},{"interface":"front","average":{"1min":18.177,"5min":22.536000000000001,"15min":23.007999999999999},"min":{"1min":13.364000000000001,"5min":9.1270000000000007,"15min":6.46},"max":{"1min":25.591000000000001,"5min":224.55500000000001,"15min":224.55500000000001},"last":9.5830000000000002}]},{"osd":6,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":13.776999999999999,"5min":20.428000000000001,"15min":23.344000000000001},"min":{"1min":8.4740000000000002,"5min":8.2129999999999992,"15min":7.8079999999999998},"max":{"1min":22.021000000000001,"5min":158.36099999999999,"15min":268.392},"last":14.996},{"interface":"front","average":{"1min":17.134,"5min":25.366,"15min":26.675999999999998},"min":{"1min":13.542999999999999,"5min":10.300000000000001,"15min":9.9299999999999997},"max":{"1min":23.768000000000001,"5min":243.40700000000001,"15min":243.40700000000001},"last":18.398}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.846,"5min":24.731999999999999,"15min":26.576000000000001},"min":{"1min":10.978,"5min":9.9939999999999998,"15min":9.2319999999999993},"max":{"1min":25.713000000000001,"5min":225.375,"15min":261.75099999999998},"last":17.609999999999999},{"interface":"front","average":{"1min":13.435,"5min":19.010999999999999,"15min":19.09},"min":{"1min":9.1319999999999997,"5min":7.181,"15min":6.7110000000000003},"max":{"1min":20.039000000000001,"5min":167.53700000000001,"15min":178.39699999999999},"last":15.609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:55:48.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.310+0000 7f2c3b089640 10 monclient: get_monmap_and_config 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.310+0000 7f2c3b089640 10 monclient: build_initial_monmap 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.310+0000 7f2c3b089640 10 monclient: monmap: 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:48.311401+0000 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:48.311401+0000 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:48.313 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:48.314 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:48.314 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:48.314 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:48.314 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 Processor -- start 2023-12-07T07:55:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 -- start start 2023-12-07T07:55:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient: init 2023-12-07T07:55:48.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c340b1b70 0x7f2c340acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): picked mon.noname-a con 0x7f2c340b1b70 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c34072890 0x7f2c341453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): picked mon.noname-b con 0x7f2c34072890 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c3413f0c0 0x7f2c3413f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): picked mon.noname-c con 0x7f2c3413f0c0 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c340ad930 con 0x7f2c340b1b70 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c340b1f50 con 0x7f2c3413f0c0 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340ad520 con 0x7f2c34072890 2023-12-07T07:55:48.320 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.311+0000 7f2c3b089640 10 monclient(hunting): authenticate will time out at 2670.690029s 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c38e25640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c340b1b70 0x7f2c340acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c33fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c34072890 0x7f2c341453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c39626640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c3413f0c0 0x7f2c3413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c38e25640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2c340b1b70 0x7f2c340acfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38560/0 (socket says 172.21.15.97:38560) 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c38e25640 1 -- 172.21.15.97:0/3593549471 learned_addr learned my addr 172.21.15.97:0/3593549471 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c38e25640 10 monclient(hunting): get_auth_request con 0x7f2c340b1b70 auth_method 0 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c38e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c38e25640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c38e25640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c39626640 10 monclient(hunting): get_auth_request con 0x7f2c3413f0c0 auth_method 0 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c39626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c39626640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c39626640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:48.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c33fff640 10 monclient(hunting): get_auth_request con 0x7f2c34072890 auth_method 0 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c33fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.312+0000 7f2c33fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c38e25640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c38e25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c38e25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c39626640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c39626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.313+0000 7f2c39626640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 10 monclient(hunting): handle_auth_done global_id 20176 payload 274 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 1 -- 172.21.15.97:0/3593549471 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c34072890 msgr2=0x7f2c341453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 1 --2- 172.21.15.97:0/3593549471 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c34072890 0x7f2c341453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 1 -- 172.21.15.97:0/3593549471 >> v2:172.21.15.97:3300/0 conn(0x7f2c340b1b70 msgr2=0x7f2c340acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 1 --2- 172.21.15.97:0/3593549471 >> v2:172.21.15.97:3300/0 conn(0x7f2c340b1b70 0x7f2c340acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 10 monclient: _finish_hunting 0 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 1 monclient: found mon.noname-c 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c39626640 1 -- 172.21.15.97:0/3593549471 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340ae1f0 con 0x7f2c3413f0c0 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.314+0000 7f2c38e25640 1 --2- 172.21.15.97:0/3593549471 >> v2:172.21.15.97:3300/0 conn(0x7f2c340b1b70 0x7f2c340acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-07T07:55:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.316+0000 7f2c39626640 1 --2- 172.21.15.97:0/3593549471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c3413f0c0 0x7f2c3413f530 secure :-1 s=READY pgs=4854 cs=0 l=1 rev1=1 crypto rx=0x7f2c340ac940 tx=0x7f2c2c02bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=446e5b7bc0b7f0fd server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 1 -- 172.21.15.97:0/3593549471 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c02cb40 con 0x7f2c3413f0c0 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: dump: 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:48.325 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: _finish_auth 0 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: _check_auth_tickets 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 1 -- 172.21.15.97:0/3593549471 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c02cca0 con 0x7f2c3413f0c0 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c3b089640 5 monclient: authenticate success, global_id 20176 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c3b089640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c3b089640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 1 -- 172.21.15.97:0/3593549471 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c034660 con 0x7f2c3413f0c0 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c3b089640 4 set_mon_vals no callback set 2023-12-07T07:55:48.326 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.320+0000 7f2c337fe640 10 monclient: dump: 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 10 monclient: shutdown 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:48.327 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 -- 172.21.15.97:0/3593549471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c3413f0c0 msgr2=0x7f2c3413f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3593549471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c3413f0c0 0x7f2c3413f530 secure :-1 s=READY pgs=4854 cs=0 l=1 rev1=1 crypto rx=0x7f2c340ac940 tx=0x7f2c2c02bd90 comp rx=0 tx=0).stop 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 -- 172.21.15.97:0/3593549471 shutdown_connections 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3593549471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c3413f0c0 0x7f2c3413f530 unknown :-1 s=CLOSED pgs=4854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3593549471 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c34072890 0x7f2c341453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3593549471 >> v2:172.21.15.97:3300/0 conn(0x7f2c340b1b70 0x7f2c340acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 -- 172.21.15.97:0/3593549471 >> 172.21.15.97:0/3593549471 conn(0x7f2c340b4d20 msgr2=0x7f2c340b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 -- 172.21.15.97:0/3593549471 shutdown_connections 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.321+0000 7f2c3b089640 1 -- 172.21.15.97:0/3593549471 wait complete. 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 10 monclient: build_initial_monmap 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 10 monclient: monmap: 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:48.328 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 1 librados: starting msgr at 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 1 librados: starting objecter 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 1 Processor -- start 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 1 -- start start 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 1 librados: setting wanted keys 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 1 librados: calling monclient init 2023-12-07T07:55:48.329 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.322+0000 7f2c3b089640 10 monclient: init 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c34072890 0x7f2c34144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): picked mon.noname-a con 0x7f2c34072890 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c340b1b70 0x7f2c34138490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): picked mon.noname-b con 0x7f2c340b1b70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c3413f0c0 0x7f2c341d9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): picked mon.noname-c con 0x7f2c3413f0c0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c34141a80 con 0x7f2c34072890 2023-12-07T07:55:48.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c341dadb0 con 0x7f2c340b1b70 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c341389d0 con 0x7f2c3413f0c0 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c3b089640 10 monclient(hunting): authenticate will time out at 2670.701608s 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c340b1b70 0x7f2c34138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c38e25640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c34072890 0x7f2c34144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.323+0000 7f2c39626640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c3413f0c0 0x7f2c341d9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c340b1b70 0x7f2c34138490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:40960/0 (socket says 172.21.15.97:40960) 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 1 -- 172.21.15.97:0/3418644184 learned_addr learned my addr 172.21.15.97:0/3418644184 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 10 monclient(hunting): get_auth_request con 0x7f2c340b1b70 auth_method 0 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:48.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c38e25640 10 monclient(hunting): get_auth_request con 0x7f2c34072890 auth_method 0 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c38e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c38e25640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c38e25640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c39626640 10 monclient(hunting): get_auth_request con 0x7f2c3413f0c0 auth_method 0 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c39626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c39626640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c39626640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c38e25640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c38e25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c38e25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:48.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c39626640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c39626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.324+0000 7f2c39626640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 10 monclient(hunting): handle_auth_done global_id 20179 payload 824 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 1 -- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c3413f0c0 msgr2=0x7f2c341d9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c3413f0c0 0x7f2c341d9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 1 -- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c34072890 msgr2=0x7f2c34144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c34072890 0x7f2c34144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 1 monclient: found mon.noname-b 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.327+0000 7f2c33fff640 1 -- 172.21.15.97:0/3418644184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c280094d0 con 0x7f2c340b1b70 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.329+0000 7f2c33fff640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c340b1b70 0x7f2c34138490 secure :-1 s=READY pgs=4855 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000e9e0 tx=0x7f2c2000eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.329+0000 7f2c317fa640 10 client.?.objecter ms_handle_connect 0x7f2c340b1b70 2023-12-07T07:55:48.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.329+0000 7f2c317fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 1 -- 172.21.15.97:0/3418644184 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20009670 con 0x7f2c340b1b70 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: dump: 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:48.337 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: _finish_auth 0 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: _check_auth_tickets 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 1 -- 172.21.15.97:0/3418644184 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2000c9d0 con 0x7f2c340b1b70 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 1 -- 172.21.15.97:0/3418644184 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000cd80 con 0x7f2c340b1b70 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c317fa640 10 monclient: dump: 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.338 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 5 monclient: authenticate success, global_id 20179 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c32ffd640 4 set_mon_vals no callback set 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 10 monclient: _renew_subs 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c009a70 con 0x7f2c340b1b70 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 10 client.20179.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 10 monclient: _renew_subs 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c341da6e0 con 0x7f2c340b1b70 2023-12-07T07:55:48.339 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.332+0000 7f2c3b089640 1 librados: init done 2023-12-07T07:55:48.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.333+0000 7f2c3b089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:48.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.333+0000 7f2c3b089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:48.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.333+0000 7f2c3b089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:48.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.333+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c34138c80 con 0x7f2c340b1b70 2023-12-07T07:55:48.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.576+0000 7f2c317fa640 1 -- 172.21.15.97:0/3418644184 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2c20013470 con 0x7f2c340b1b70 2023-12-07T07:55:48.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.576+0000 7f2c317fa640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2c1806c820 0x7f2c1806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:48.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.576+0000 7f2c38e25640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2c1806c820 0x7f2c1806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:48.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.577+0000 7f2c38e25640 10 monclient: get_auth_request con 0x7f2c1806c820 auth_method 0 2023-12-07T07:55:48.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.577+0000 7f2c317fa640 10 client.20179.objecter ms_handle_connect 0x7f2c1806c820 2023-12-07T07:55:48.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.577+0000 7f2c38e25640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2c1806c820 0x7f2c1806ece0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006ca0 tx=0x7f2c28009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.579+0000 7f2c317fa640 1 -- 172.21.15.97:0/3418644184 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f2c20058d80 con 0x7f2c340b1b70 2023-12-07T07:55:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.579+0000 7f2c317fa640 10 client.20179.objecter ms_dispatch 0x7f2c34000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.579+0000 7f2c317fa640 3 client.20179.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.579+0000 7f2c317fa640 3 client.20179.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.579+0000 7f2c317fa640 20 client.20179.objecter dump_active .. 0 homeless 2023-12-07T07:55:48.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.589+0000 7f2c317fa640 1 -- 172.21.15.97:0/3418644184 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c200550c0 con 0x7f2c340b1b70 2023-12-07T07:55:48.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.589+0000 7f2c317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:48.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.589+0000 7f2c317fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:48.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.760+0000 7f2c3b089640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:55:48.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.760+0000 7f2c3b089640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:55:48.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.760+0000 7f2c3b089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:48.763 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.760+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c34075b60 con 0x7f2c340b1b70 2023-12-07T07:55:48.798 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.796+0000 7f2c317fa640 1 -- 172.21.15.97:0/3418644184 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v920) v1 ==== 74+0+17480 (secure 0 0 0) 0x7f2c20017090 con 0x7f2c340b1b70 2023-12-07T07:55:48.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.796+0000 7f2c317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-07T07:55:48.799 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.796+0000 7f2c317fa640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:55:48.799 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:55:48.799 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":920,"fsid":"f040f620-331a-4806-a30a-a56e2cce06a5","created":"2023-12-07T07:32:45.345822+0000","modified":"2023-12-07T07:55:45.944651+0000","last_up_change":"2023-12-07T07:33:33.953336+0000","last_in_change":"2023-12-07T07:32:50.728917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":99,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-07T07:33:30.063715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T07:33:40.220751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-12-07T07:34:40.673563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb9fa9f-1af4-46cd-af57-26660874af7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6819","nonce":2798215727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6821","nonce":2798215727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6825","nonce":2798215727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2798215727},{"type":"v1","addr":"172.21.15.97:6823","nonce":2798215727}]},"public_addr":"172.21.15.97:6819/2798215727","cluster_addr":"172.21.15.97:6821/2798215727","heartbeat_back_addr":"172.21.15.97:6825/2798215727","heartbeat_front_addr":"172.21.15.97:6823/2798215727","state":["exists","up"]},{"osd":1,"uuid":"ab4dbc4f-fed6-4ad2-934d-76bfb487ddeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6827","nonce":138440183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6829","nonce":138440183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6833","nonce":138440183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":138440183},{"type":"v1","addr":"172.21.15.97:6831","nonce":138440183}]},"public_addr":"172.21.15.97:6827/138440183","cluster_addr":"172.21.15.97:6829/138440183","heartbeat_back_addr":"172.21.15.97:6833/138440183","heartbeat_front_addr":"172.21.15.97:6831/138440183","state":["exists","up"]},{"osd":2,"uuid":"2e035cab-ae59-4d53-b46e-867d865bfc46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6803","nonce":1673674356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6805","nonce":1673674356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6809","nonce":1673674356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":1673674356},{"type":"v1","addr":"172.21.15.97:6807","nonce":1673674356}]},"public_addr":"172.21.15.97:6803/1673674356","cluster_addr":"172.21.15.97:6805/1673674356","heartbeat_back_addr":"172.21.15.97:6809/1673674356","heartbeat_front_addr":"172.21.15.97:6807/1673674356","state":["exists","up"]},{"osd":3,"uuid":"e1148e29-3789-483a-8463-ea1e1f964b39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6811","nonce":929626392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6813","nonce":929626392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6817","nonce":929626392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":929626392},{"type":"v1","addr":"172.21.15.97:6815","nonce":929626392}]},"public_addr":"172.21.15.97:6811/929626392","cluster_addr":"172.21.15.97:6813/929626392","heartbeat_back_addr":"172.21.15.97:6817/929626392","heartbeat_front_addr":"172.21.15.97:6815/929626392","state":["exists","up"]},{"osd":4,"uuid":"f525c573-e372-4e52-ab23-48a4a0905272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":909,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6809","nonce":1781054720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6811","nonce":1781054720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6815","nonce":1781054720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1781054720},{"type":"v1","addr":"172.21.15.154:6813","nonce":1781054720}]},"public_addr":"172.21.15.154:6809/1781054720","cluster_addr":"172.21.15.154:6811/1781054720","heartbeat_back_addr":"172.21.15.154:6815/1781054720","heartbeat_front_addr":"172.21.15.154:6813/1781054720","state":["exists","up"]},{"osd":5,"uuid":"4dfbc265-0117-4a4e-95f8-433766bd4aab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6801","nonce":4174110211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6803","nonce":4174110211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6807","nonce":4174110211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4174110211},{"type":"v1","addr":"172.21.15.154:6805","nonce":4174110211}]},"public_addr":"172.21.15.154:6801/4174110211","cluster_addr":"172.21.15.154:6803/4174110211","heartbeat_back_addr":"172.21.15.154:6807/4174110211","heartbeat_front_addr":"172.21.15.154:6805/4174110211","state":["exists","up"]},{"osd":6,"uuid":"9f9c29c8-a3c3-41ba-8bd7-51ae208cb2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6825","nonce":1615374996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6827","nonce":1615374996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6831","nonce":1615374996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1615374996},{"type":"v1","addr":"172.21.15.154:6829","nonce":1615374996}]},"public_addr":"172.21.15.154:6825/1615374996","cluster_addr":"172.21.15.154:6827/1615374996","heartbeat_back_addr":"172.21.15.154:6831/1615374996","heartbeat_front_addr":"172.21.15.154:6829/1615374996","state":["exists","up"]},{"osd":7,"uuid":"f35f7cd5-7ce9-4336-a36d-836f1cff51c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":914,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6817","nonce":1161771449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6819","nonce":1161771449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6823","nonce":1161771449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1161771449},{"type":"v1","addr":"172.21.15.154:6821","nonce":1161771449}]},"public_addr":"172.21.15.154:6817/1161771449","cluster_addr":"172.21.15.154:6819/1161771449","heartbeat_back_addr":"172.21.15.154:6823/1161771449","heartbeat_front_addr":"172.21.15.154:6821/1161771449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.269586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.927364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.534718+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:30.623805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.804774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.252422+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:28.225222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T07:33:27.956517+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":9,"length":2}]},{"pool":14,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":66,"snaps":[{"begin":11,"length":2}]},{"pool":88,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":100,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":101,"snaps":[{"begin":2,"length":1}]},{"pool":104,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":113,"snaps":[{"begin":4,"length":3}]},{"pool":116,"snaps":[{"begin":2,"length":1},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":176,"snaps":[{"begin":12,"length":1}]},{"pool":186,"snaps":[{"begin":14,"length":1}]},{"pool":191,"snaps":[{"begin":17,"length":1}]},{"pool":193,"snaps":[{"begin":19,"length":2}]},{"pool":197,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 10 librados: watch_flush enter 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 10 librados: watch_flush exit 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 20 client.20179.objecter shutdown clearing up homeless session... 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 10 client.20179.objecter successfully canceled tick 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2c1806c820 msgr2=0x7f2c1806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2c1806c820 0x7f2c1806ece0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006ca0 tx=0x7f2c28009240 comp rx=0 tx=0).stop 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 10 monclient: shutdown 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c340b1b70 msgr2=0x7f2c34138490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:48.801 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c340b1b70 0x7f2c34138490 secure :-1 s=READY pgs=4855 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000e9e0 tx=0x7f2c2000eea0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 shutdown_connections 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2c1806c820 0x7f2c1806ece0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2c3413f0c0 0x7f2c341d9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c340b1b70 0x7f2c34138490 unknown :-1 s=CLOSED pgs=4855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 --2- 172.21.15.97:0/3418644184 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2c34072890 0x7f2c34144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 >> 172.21.15.97:0/3418644184 conn(0x7f2c340b4d20 msgr2=0x7f2c3413bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 shutdown_connections 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 -- 172.21.15.97:0/3418644184 wait complete. 2023-12-07T07:55:48.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:48.798+0000 7f2c3b089640 1 librados: shutdown 2023-12-07T07:55:49.813 INFO:tasks.ceph:Scrubbing osd.0 2023-12-07T07:55:49.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.917+0000 7f10597d0640 10 monclient: get_monmap_and_config 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.917+0000 7f10597d0640 10 monclient: build_initial_monmap 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.917+0000 7f10597d0640 10 monclient: monmap: 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:49.918785+0000 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:49.918785+0000 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:49.920 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:49.921 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:49.921 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.918+0000 7f10597d0640 1 Processor -- start 2023-12-07T07:55:49.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.918+0000 7f10597d0640 1 -- start start 2023-12-07T07:55:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.918+0000 7f10597d0640 10 monclient: init 2023-12-07T07:55:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.918+0000 7f10597d0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.918+0000 7f10597d0640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-12-07T07:55:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10540a88a0 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f105413f7c0 0x7f10540a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 10 monclient(hunting): picked mon.noname-a con 0x7f105413f7c0 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1054145480 0x7f1054145910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 10 monclient(hunting): picked mon.noname-b con 0x7f1054145480 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10540b17d0 con 0x7f10540a88a0 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10540aac60 con 0x7f105413f7c0 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540b1560 con 0x7f1054145480 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10597d0640 10 monclient(hunting): authenticate will time out at 2672.297372s 2023-12-07T07:55:49.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.919+0000 7f10537fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1054145480 0x7f1054145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f104bfff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f105413f7c0 0x7f10540a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1054145480 0x7f1054145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:59884/0 (socket says 172.21.15.97:59884) 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 1 -- 172.21.15.97:0/2961744987 learned_addr learned my addr 172.21.15.97:0/2961744987 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 10 monclient(hunting): get_auth_request con 0x7f1054145480 auth_method 0 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.920+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.921+0000 7f104bfff640 10 monclient(hunting): get_auth_request con 0x7f105413f7c0 auth_method 0 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.921+0000 7f104bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:49.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.921+0000 7f104bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:49.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.921+0000 7f104bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:49.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 10 monclient(hunting): handle_auth_done global_id 17660 payload 274 2023-12-07T07:55:49.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 1 -- 172.21.15.97:0/2961744987 >> v2:172.21.15.97:3300/0 conn(0x7f105413f7c0 msgr2=0x7f10540a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:49.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 1 --2- 172.21.15.97:0/2961744987 >> v2:172.21.15.97:3300/0 conn(0x7f105413f7c0 0x7f10540a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:49.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 1 -- 172.21.15.97:0/2961744987 >> v1:172.21.15.97:6789/0 conn(0x7f10540a88a0 legacy=0x7f105413f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:49.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 10 monclient: _finish_hunting 0 2023-12-07T07:55:49.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 1 monclient: found mon.noname-b 2023-12-07T07:55:49.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:49.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:55:49.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.922+0000 7f10537fe640 1 -- 172.21.15.97:0/2961744987 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540a97d0 con 0x7f1054145480 2023-12-07T07:55:49.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.924+0000 7f10537fe640 1 --2- 172.21.15.97:0/2961744987 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1054145480 0x7f1054145910 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f104c009e50 tx=0x7f104c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4c17d46a30c0011 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 1 -- 172.21.15.97:0/2961744987 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c02ca70 con 0x7f1054145480 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: dump: 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:49.932 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: _finish_auth 0 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: _check_auth_tickets 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 1 -- 172.21.15.97:0/2961744987 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c02cbd0 con 0x7f1054145480 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 5 monclient: authenticate success, global_id 17660 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:49.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 4 set_mon_vals no callback set 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 1 -- 172.21.15.97:0/2961744987 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c0343f0 con 0x7f1054145480 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10527fc640 10 monclient: dump: 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:49.934 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 10 monclient: shutdown 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 -- 172.21.15.97:0/2961744987 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1054145480 msgr2=0x7f1054145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 --2- 172.21.15.97:0/2961744987 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1054145480 0x7f1054145910 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f104c009e50 tx=0x7f104c02bed0 comp rx=0 tx=0).stop 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 -- 172.21.15.97:0/2961744987 shutdown_connections 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 --2- 172.21.15.97:0/2961744987 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1054145480 0x7f1054145910 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 --2- 172.21.15.97:0/2961744987 >> v2:172.21.15.97:3300/0 conn(0x7f105413f7c0 0x7f10540a8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 -- 172.21.15.97:0/2961744987 >> 172.21.15.97:0/2961744987 conn(0x7f10540619a0 msgr2=0x7f10540b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 -- 172.21.15.97:0/2961744987 shutdown_connections 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.928+0000 7f10597d0640 1 -- 172.21.15.97:0/2961744987 wait complete. 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 10 monclient: build_initial_monmap 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 10 monclient: monmap: 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:49.935 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 1 librados: starting msgr at 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 1 librados: starting objecter 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 1 Processor -- start 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 1 -- start start 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 1 librados: setting wanted keys 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 1 librados: calling monclient init 2023-12-07T07:55:49.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.929+0000 7f10597d0640 10 monclient: init 2023-12-07T07:55:49.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f10540a88a0 0x7f1054139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): picked mon.noname-c con 0x7f10540a88a0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f105413f7c0 0x7f10541da540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): picked mon.noname-b con 0x7f105413f7c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1054145480 0x7f10541daa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): picked mon.noname-a con 0x7f1054145480 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1054136eb0 con 0x7f1054145480 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10541dafc0 con 0x7f105413f7c0 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1054136990 con 0x7f10540a88a0 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f10597d0640 10 monclient(hunting): authenticate will time out at 2672.308669s 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.930+0000 7f1052ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f10540a88a0 0x7f1054139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f105413f7c0 0x7f10541da540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f10540a88a0 0x7f1054139ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:59898/0 (socket says 172.21.15.97:59898) 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 1 -- 172.21.15.97:0/1873512503 learned_addr learned my addr 172.21.15.97:0/1873512503 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 10 monclient(hunting): get_auth_request con 0x7f10540a88a0 auth_method 0 2023-12-07T07:55:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f10537fe640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1054145480 0x7f10541daa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 10 monclient(hunting): get_auth_request con 0x7f105413f7c0 auth_method 0 2023-12-07T07:55:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:49.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f10537fe640 10 monclient(hunting): get_auth_request con 0x7f1054145480 auth_method 0 2023-12-07T07:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f10537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f10537fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f10537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f1052ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.931+0000 7f104bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.932+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.932+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.932+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 10 monclient(hunting): handle_auth_done global_id 17663 payload 824 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 1 -- 172.21.15.97:0/1873512503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f105413f7c0 msgr2=0x7f10541da540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f105413f7c0 0x7f10541da540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 1 -- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1054145480 msgr2=0x7f10541daa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1054145480 0x7f10541daa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 10 monclient: _finish_hunting 0 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 1 monclient: found mon.noname-c 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.934+0000 7f1052ffd640 1 -- 172.21.15.97:0/1873512503 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c009b30 con 0x7f10540a88a0 2023-12-07T07:55:49.945 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.936+0000 7f1052ffd640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f10540a88a0 0x7f1054139ad0 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f104400c450 tx=0x7f104400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:49.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.936+0000 7f104b7fe640 10 client.?.objecter ms_handle_connect 0x7f10540a88a0 2023-12-07T07:55:49.946 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.936+0000 7f104b7fe640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440052d0 con 0x7f10540a88a0 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: dump: 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.948 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: _finish_auth 0 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: _check_auth_tickets 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1044005430 con 0x7f10540a88a0 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044012870 con 0x7f10540a88a0 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 5 monclient: authenticate success, global_id 17663 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 10 monclient: _renew_subs 2023-12-07T07:55:49.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105413fbd0 con 0x7f10540a88a0 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f1051ffb640 4 set_mon_vals no callback set 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f104b7fe640 10 monclient: dump: 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:49.950 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 10 client.17663.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 10 monclient: _renew_subs 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10541372b0 con 0x7f10540a88a0 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.940+0000 7f10597d0640 1 librados: init done 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.941+0000 7f10597d0640 10 client.17663.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.941+0000 7f10597d0640 15 client.17663.objecter _session_command_op_assign -1 1 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.941+0000 7f10597d0640 20 client.17663.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.941+0000 7f10597d0640 10 client.17663.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.941+0000 7f10597d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:49.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.941+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10540678d0 con 0x7f10540a88a0 2023-12-07T07:55:49.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.960+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f10440129d0 con 0x7f10540a88a0 2023-12-07T07:55:49.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.960+0000 7f104b7fe640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f103406c7d0 0x7f103406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:49.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.960+0000 7f104bfff640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f103406c7d0 0x7f103406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:49.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.960+0000 7f104bfff640 10 monclient: get_auth_request con 0x7f103406c7d0 auth_method 0 2023-12-07T07:55:49.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.961+0000 7f104b7fe640 10 client.17663.objecter ms_handle_connect 0x7f103406c7d0 2023-12-07T07:55:49.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.961+0000 7f104bfff640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f103406c7d0 0x7f103406ec90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f105413ac80 tx=0x7f103c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f1044058860 con 0x7f10540a88a0 2023-12-07T07:55:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 10 client.17663.objecter ms_dispatch 0x7f1054000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 3 client.17663.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:49.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 3 client.17663.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 10 client.17663.objecter checking command 1 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1034074100 0x7f10340765a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 20 client.17663.objecter _get_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 20 client.17663.objecter put_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 15 client.17663.objecter _session_command_op_remove -1 1 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 20 client.17663.objecter _get_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 20 client.17663.objecter get_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 15 client.17663.objecter _session_command_op_assign 0 1 2023-12-07T07:55:49.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 20 client.17663.objecter put_session s=0x7f1034073a30 osd=0 4 2023-12-07T07:55:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 10 client.17663.objecter _send_command 1 2023-12-07T07:55:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1034076c70 con 0x7f1034074100 2023-12-07T07:55:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 20 client.17663.objecter dump_active .. 0 homeless 2023-12-07T07:55:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10440011e0 con 0x7f10540a88a0 2023-12-07T07:55:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.963+0000 7f104b7fe640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:49.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.964+0000 7f10537fe640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1034074100 0x7f10340765a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.964+0000 7f10537fe640 10 monclient: get_auth_request con 0x7f1034074100 auth_method 0 2023-12-07T07:55:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.967+0000 7f104b7fe640 10 client.17663.objecter ms_handle_connect 0x7f1034074100 2023-12-07T07:55:49.969 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.967+0000 7f10537fe640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1034074100 0x7f10340765a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:49.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.991+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== osd.0 v2:172.21.15.97:6818/2798215727 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1034076c70 con 0x7f1034074100 2023-12-07T07:55:49.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.991+0000 7f104b7fe640 10 client.17663.objecter ms_dispatch 0x7f1054000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:49.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.991+0000 7f104b7fe640 10 client.17663.objecter _finish_command 1 = system:0 2023-12-07T07:55:49.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.991+0000 7f104b7fe640 20 client.17663.objecter put_session s=0x7f1034073a30 osd=0 4 2023-12-07T07:55:49.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:49.991+0000 7f104b7fe640 15 client.17663.objecter _session_command_op_remove 0 1 2023-12-07T07:55:50.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 10 client.17663.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 15 client.17663.objecter _session_command_op_assign -1 2 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 20 client.17663.objecter _get_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 20 client.17663.objecter put_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 20 client.17663.objecter _get_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 15 client.17663.objecter _session_command_op_remove -1 2 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 20 client.17663.objecter get_session s=0x7f1034073a30 osd=0 3 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 15 client.17663.objecter _session_command_op_assign 0 2 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 20 client.17663.objecter put_session s=0x7f1034073a30 osd=0 4 2023-12-07T07:55:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 10 client.17663.objecter _send_command 2 2023-12-07T07:55:50.008 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.004+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 --> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1054136bb0 con 0x7f1034074100 2023-12-07T07:55:50.020 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f104b7fe640 1 -- 172.21.15.97:0/1873512503 <== osd.0 v2:172.21.15.97:6818/2798215727 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1054136bb0 con 0x7f1034074100 2023-12-07T07:55:50.020 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f104b7fe640 10 client.17663.objecter ms_dispatch 0x7f1054000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:50.020 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f104b7fe640 10 client.17663.objecter _finish_command 2 = system:0 2023-12-07T07:55:50.020 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f104b7fe640 20 client.17663.objecter put_session s=0x7f1034073a30 osd=0 4 2023-12-07T07:55:50.020 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f104b7fe640 15 client.17663.objecter _session_command_op_remove 0 2 2023-12-07T07:55:50.021 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:50.021 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:50.021 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 10 librados: watch_flush enter 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 10 librados: watch_flush exit 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 10 client.17663.objecter close_session for osd.0 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1034074100 msgr2=0x7f10340765a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6818/2798215727,v1:172.21.15.97:6819/2798215727] conn(0x7f1034074100 0x7f10340765a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 20 client.17663.objecter put_session s=0x7f1034073a30 osd=0 1 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 20 client.17663.objecter shutdown clearing up homeless session... 2023-12-07T07:55:50.023 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 10 client.17663.objecter successfully canceled tick 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f103406c7d0 msgr2=0x7f103406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f103406c7d0 0x7f103406ec90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f105413ac80 tx=0x7f103c008040 comp rx=0 tx=0).stop 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 10 monclient: shutdown 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f10540a88a0 msgr2=0x7f1054139ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 --2- 172.21.15.97:0/1873512503 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f10540a88a0 0x7f1054139ad0 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f104400c450 tx=0x7f104400c910 comp rx=0 tx=0).stop 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10537fe640 1 -- 172.21.15.97:0/1873512503 reap_dead start 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 shutdown_connections 2023-12-07T07:55:50.024 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 >> 172.21.15.97:0/1873512503 conn(0x7f10540619a0 msgr2=0x7f105413b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:50.025 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 shutdown_connections 2023-12-07T07:55:50.025 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 -- 172.21.15.97:0/1873512503 wait complete. 2023-12-07T07:55:50.025 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.017+0000 7f10597d0640 1 librados: shutdown 2023-12-07T07:55:50.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-07T07:55:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.127+0000 7f6478cb0640 10 monclient: get_monmap_and_config 2023-12-07T07:55:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.128+0000 7f6478cb0640 10 monclient: build_initial_monmap 2023-12-07T07:55:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.128+0000 7f6478cb0640 10 monclient: monmap: 2023-12-07T07:55:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:50.130 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:50.129078+0000 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:50.129078+0000 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:50.131 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.132 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.128+0000 7f6478cb0640 1 Processor -- start 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.128+0000 7f6478cb0640 1 -- start start 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.128+0000 7f6478cb0640 10 monclient: init 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64740a8910 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64740a9b70 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f647413f370 0x7f64741426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient(hunting): picked mon.noname-a con 0x7f647413f370 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64740a8cf0 con 0x7f64740a9b70 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64740aabd0 con 0x7f64740a8910 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64740aaeb0 con 0x7f647413f370 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.129+0000 7f6478cb0640 10 monclient(hunting): authenticate will time out at 2672.507678s 2023-12-07T07:55:50.133 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.130+0000 7f647259c640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f64740a8910 0x7f647413ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:48594/0 (socket says 172.21.15.97:48594) 2023-12-07T07:55:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.130+0000 7f647259c640 1 -- 172.21.15.97:0/492782140 learned_addr learned my addr 172.21.15.97:0/492782140 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.130+0000 7f6472d9d640 1 --2- 172.21.15.97:0/492782140 >> v2:172.21.15.97:3300/0 conn(0x7f647413f370 0x7f64741426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.130+0000 7f6472d9d640 10 monclient(hunting): get_auth_request con 0x7f647413f370 auth_method 0 2023-12-07T07:55:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.130+0000 7f6472d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.130+0000 7f6472d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.130+0000 7f6472d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.134 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.131+0000 7f6472d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.131+0000 7f6472d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.131+0000 7f6472d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.132+0000 7f647159a640 1 -- 172.21.15.97:0/492782140 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294502169 0 0) 0x7f64740aabd0 con 0x7f64740a8910 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.132+0000 7f647159a640 10 monclient(hunting): _negotiate 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.132+0000 7f647159a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.132+0000 7f647159a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.132+0000 7f647159a640 10 monclient(hunting): my global_id is 20185 2023-12-07T07:55:50.135 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.132+0000 7f647159a640 1 -- 172.21.15.97:0/492782140 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645c003520 con 0x7f64740a8910 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 10 monclient(hunting): handle_auth_done global_id 16125 payload 274 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 1 -- 172.21.15.97:0/492782140 >> v1:172.21.15.154:6789/0 conn(0x7f64740a8910 legacy=0x7f647413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 1 -- 172.21.15.97:0/492782140 >> v1:172.21.15.97:6789/0 conn(0x7f64740a9b70 legacy=0x7f6474141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 10 monclient: _finish_hunting 0 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 1 monclient: found mon.noname-a 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:50.136 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.133+0000 7f6472d9d640 1 -- 172.21.15.97:0/492782140 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64740a9720 con 0x7f647413f370 2023-12-07T07:55:50.139 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.136+0000 7f6472d9d640 1 --2- 172.21.15.97:0/492782140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413f370 0x7f64741426a0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f646c0097a0 tx=0x7f646c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f56ac22e9261a1a server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 1 -- 172.21.15.97:0/492782140 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c030610 con 0x7f647413f370 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: dump: 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.149 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: _finish_auth 0 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: _check_auth_tickets 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 1 -- 172.21.15.97:0/492782140 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c030770 con 0x7f647413f370 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f6478cb0640 5 monclient: authenticate success, global_id 16125 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f6478cb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f6478cb0640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f6478cb0640 4 set_mon_vals no callback set 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 1 -- 172.21.15.97:0/492782140 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c030b00 con 0x7f647413f370 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.146+0000 7f647159a640 10 monclient: dump: 2023-12-07T07:55:50.150 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.151 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 10 monclient: shutdown 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 1 -- 172.21.15.97:0/492782140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413f370 msgr2=0x7f64741426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 1 --2- 172.21.15.97:0/492782140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413f370 0x7f64741426a0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f646c0097a0 tx=0x7f646c02b960 comp rx=0 tx=0).stop 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 1 -- 172.21.15.97:0/492782140 shutdown_connections 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 1 --2- 172.21.15.97:0/492782140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413f370 0x7f64741426a0 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 1 -- 172.21.15.97:0/492782140 >> 172.21.15.97:0/492782140 conn(0x7f64740b7780 msgr2=0x7f64740b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:50.154 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 1 -- 172.21.15.97:0/492782140 shutdown_connections 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.147+0000 7f6478cb0640 1 -- 172.21.15.97:0/492782140 wait complete. 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 10 monclient: build_initial_monmap 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 10 monclient: monmap: 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 1 librados: starting msgr at 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 1 librados: starting objecter 2023-12-07T07:55:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 1 Processor -- start 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 1 -- start start 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 1 librados: setting wanted keys 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 1 librados: calling monclient init 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.148+0000 7f6478cb0640 10 monclient: init 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f64740a8910 0x7f64740ac340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): picked mon.noname-c con 0x7f64740a8910 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647413f370 0x7f647413ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): picked mon.noname-b con 0x7f647413f370 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413b280 0x7f6474132690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): picked mon.noname-a con 0x7f647413b280 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64740abd20 con 0x7f647413b280 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.156 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647413a090 con 0x7f647413f370 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6474139360 con 0x7f64740a8910 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6478cb0640 10 monclient(hunting): authenticate will time out at 2672.527435s 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f647259c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f64740a8910 0x7f64740ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6471d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647413f370 0x7f647413ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6471d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647413f370 0x7f647413ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:40988/0 (socket says 172.21.15.97:40988) 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6471d9b640 1 -- 172.21.15.97:0/2809600832 learned_addr learned my addr 172.21.15.97:0/2809600832 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6471d9b640 10 monclient(hunting): get_auth_request con 0x7f647413f370 auth_method 0 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6471d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6471d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.149+0000 7f6471d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f647259c640 10 monclient(hunting): get_auth_request con 0x7f64740a8910 auth_method 0 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f647259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.157 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f647259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f647259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f6471d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f6471d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f6471d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f647259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f647259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.158 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.150+0000 7f647259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.152+0000 7f6471d9b640 10 monclient(hunting): handle_auth_done global_id 20188 payload 824 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.152+0000 7f6471d9b640 1 -- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f64740a8910 msgr2=0x7f64740ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.152+0000 7f6471d9b640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f64740a8910 0x7f64740ac340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.153+0000 7f6471d9b640 1 -- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413b280 msgr2=0x7f6474132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.153+0000 7f6471d9b640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413b280 0x7f6474132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.153+0000 7f6471d9b640 10 monclient: _finish_hunting 0 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.153+0000 7f6471d9b640 1 monclient: found mon.noname-b 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.153+0000 7f6471d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.153+0000 7f6471d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.153+0000 7f6471d9b640 1 -- 172.21.15.97:0/2809600832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c009400 con 0x7f647413f370 2023-12-07T07:55:50.160 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.155+0000 7f6471d9b640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647413f370 0x7f647413ad40 secure :-1 s=READY pgs=4857 cs=0 l=1 rev1=1 crypto rx=0x7f646800c450 tx=0x7f646800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.160 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.155+0000 7f6462ffd640 10 client.?.objecter ms_handle_connect 0x7f647413f370 2023-12-07T07:55:50.160 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.155+0000 7f6462ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 1 -- 172.21.15.97:0/2809600832 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680052d0 con 0x7f647413f370 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: dump: 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: _finish_auth 0 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 1 -- 172.21.15.97:0/2809600832 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468005430 con 0x7f647413f370 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 5 monclient: authenticate success, global_id 20188 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 10 monclient: _renew_subs 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64740ac880 con 0x7f647413f370 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 1 -- 172.21.15.97:0/2809600832 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468012870 con 0x7f647413f370 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6462ffd640 10 monclient: dump: 2023-12-07T07:55:50.163 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6470d99640 4 set_mon_vals no callback set 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 10 client.20188.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 10 monclient: _renew_subs 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6474132bd0 con 0x7f647413f370 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.158+0000 7f6478cb0640 1 librados: init done 2023-12-07T07:55:50.164 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.159+0000 7f6478cb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:50.165 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.159+0000 7f6478cb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:50.165 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.159+0000 7f6478cb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.165 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.159+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6474075b60 con 0x7f647413f370 2023-12-07T07:55:50.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.178+0000 7f6462ffd640 1 -- 172.21.15.97:0/2809600832 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f64680129d0 con 0x7f647413f370 2023-12-07T07:55:50.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.178+0000 7f6462ffd640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f645406c800 0x7f645406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.179+0000 7f647259c640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f645406c800 0x7f645406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.179+0000 7f647259c640 10 monclient: get_auth_request con 0x7f645406c800 auth_method 0 2023-12-07T07:55:50.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.179+0000 7f647259c640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f645406c800 0x7f645406ecc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6464001560 tx=0x7f6464001a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.182 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.179+0000 7f6462ffd640 10 client.20188.objecter ms_handle_connect 0x7f645406c800 2023-12-07T07:55:50.183 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.181+0000 7f6462ffd640 1 -- 172.21.15.97:0/2809600832 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f6468058ff0 con 0x7f647413f370 2023-12-07T07:55:50.183 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.181+0000 7f6462ffd640 10 client.20188.objecter ms_dispatch 0x7f6474000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:50.183 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.181+0000 7f6462ffd640 3 client.20188.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:50.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.181+0000 7f6462ffd640 3 client.20188.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:50.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.181+0000 7f6462ffd640 20 client.20188.objecter dump_active .. 0 homeless 2023-12-07T07:55:50.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.413+0000 7f6462ffd640 1 -- 172.21.15.97:0/2809600832 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6468054c30 con 0x7f647413f370 2023-12-07T07:55:50.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.413+0000 7f6462ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:50.416 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.413+0000 7f6462ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:50.587 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.585+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f6474142be0 con 0x7f645406c800 2023-12-07T07:55:50.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.585+0000 7f6462ffd640 1 -- 172.21.15.97:0/2809600832 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6464001f60 con 0x7f645406c800 2023-12-07T07:55:50.588 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 0 to deep-scrub 2023-12-07T07:55:50.590 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 10 librados: watch_flush enter 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 10 librados: watch_flush exit 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 20 client.20188.objecter shutdown clearing up homeless session... 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 10 client.20188.objecter successfully canceled tick 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f645406c800 msgr2=0x7f645406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f645406c800 0x7f645406ecc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6464001560 tx=0x7f6464001a90 comp rx=0 tx=0).stop 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 10 monclient: shutdown 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647413f370 msgr2=0x7f647413ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647413f370 0x7f647413ad40 secure :-1 s=READY pgs=4857 cs=0 l=1 rev1=1 crypto rx=0x7f646800c450 tx=0x7f646800c910 comp rx=0 tx=0).stop 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 shutdown_connections 2023-12-07T07:55:50.591 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f645406c800 0x7f645406ecc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f647413b280 0x7f6474132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647413f370 0x7f647413ad40 unknown :-1 s=CLOSED pgs=4857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 --2- 172.21.15.97:0/2809600832 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f64740a8910 0x7f64740ac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 >> 172.21.15.97:0/2809600832 conn(0x7f64740b7780 msgr2=0x7f64740b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 shutdown_connections 2023-12-07T07:55:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 -- 172.21.15.97:0/2809600832 wait complete. 2023-12-07T07:55:50.592 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.587+0000 7f6478cb0640 1 librados: shutdown 2023-12-07T07:55:50.602 INFO:tasks.ceph:Scrubbing osd.1 2023-12-07T07:55:50.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:50.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.703+0000 7f0b8e094640 10 monclient: get_monmap_and_config 2023-12-07T07:55:50.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.703+0000 7f0b8e094640 10 monclient: build_initial_monmap 2023-12-07T07:55:50.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.703+0000 7f0b8e094640 10 monclient: monmap: 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:50.704257+0000 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:50.704257+0000 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:50.706 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 1 Processor -- start 2023-12-07T07:55:50.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 1 -- start start 2023-12-07T07:55:50.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient: init 2023-12-07T07:55:50.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:50.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-12-07T07:55:50.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0b88145840 0x7f0b88146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient(hunting): picked mon.noname-c con 0x7f0b88145840 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b880b20f0 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0b880b2520 0x7f0b8814c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient(hunting): picked mon.noname-a con 0x7f0b880b2520 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b880aa850 con 0x7f0b880b20f0 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b881465c0 con 0x7f0b880b2520 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b88145c20 con 0x7f0b88145840 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.704+0000 7f0b8e094640 10 monclient(hunting): authenticate will time out at 2673.083045s 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b877fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0b88145840 0x7f0b88146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b87fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0b880b2520 0x7f0b8814c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b7effd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0b880b20f0 0x7f0b8813f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47028/0 (socket says 172.21.15.97:47028) 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b877fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0b88145840 0x7f0b88146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:42022/0 (socket says 172.21.15.97:42022) 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b7effd640 1 -- 172.21.15.97:0/2571550642 learned_addr learned my addr 172.21.15.97:0/2571550642 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b877fe640 10 monclient(hunting): get_auth_request con 0x7f0b88145840 auth_method 0 2023-12-07T07:55:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b877fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.705+0000 7f0b877fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.706+0000 7f0b877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.706+0000 7f0b877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.706+0000 7f0b877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.706+0000 7f0b87fff640 10 monclient(hunting): get_auth_request con 0x7f0b880b2520 auth_method 0 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.706+0000 7f0b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.706+0000 7f0b87fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.706+0000 7f0b87fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 10 monclient(hunting): handle_auth_done global_id 20191 payload 274 2023-12-07T07:55:50.713 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 1 -- 172.21.15.97:0/2571550642 >> v2:172.21.15.97:3300/0 conn(0x7f0b880b2520 msgr2=0x7f0b8814c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 1 --2- 172.21.15.97:0/2571550642 >> v2:172.21.15.97:3300/0 conn(0x7f0b880b2520 0x7f0b8814c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 1 -- 172.21.15.97:0/2571550642 >> v1:172.21.15.97:6789/0 conn(0x7f0b880b20f0 legacy=0x7f0b8813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 10 monclient: _finish_hunting 0 2023-12-07T07:55:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 1 monclient: found mon.noname-c 2023-12-07T07:55:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.707+0000 7f0b877fe640 1 -- 172.21.15.97:0/2571550642 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b880a9260 con 0x7f0b88145840 2023-12-07T07:55:50.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.709+0000 7f0b877fe640 1 --2- 172.21.15.97:0/2571550642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 0x7f0b88146080 secure :-1 s=READY pgs=4858 cs=0 l=1 rev1=1 crypto rx=0x7f0b880b7600 tx=0x7f0b7802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bdae47c1032e4a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 1 -- 172.21.15.97:0/2571550642 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7802e070 con 0x7f0b88145840 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: dump: 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.716 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.717 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: _finish_auth 0 2023-12-07T07:55:50.717 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 1 -- 172.21.15.97:0/2571550642 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b780315a0 con 0x7f0b88145840 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 1 -- 172.21.15.97:0/2571550642 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78031930 con 0x7f0b88145840 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b86ffd640 10 monclient: dump: 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.718 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b8e094640 5 monclient: authenticate success, global_id 20191 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b8e094640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:50.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.713+0000 7f0b8e094640 4 set_mon_vals no callback set 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 10 monclient: shutdown 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 -- 172.21.15.97:0/2571550642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 msgr2=0x7f0b88146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 --2- 172.21.15.97:0/2571550642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 0x7f0b88146080 secure :-1 s=READY pgs=4858 cs=0 l=1 rev1=1 crypto rx=0x7f0b880b7600 tx=0x7f0b7802b840 comp rx=0 tx=0).stop 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 -- 172.21.15.97:0/2571550642 shutdown_connections 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 --2- 172.21.15.97:0/2571550642 >> v2:172.21.15.97:3300/0 conn(0x7f0b880b2520 0x7f0b8814c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 --2- 172.21.15.97:0/2571550642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 0x7f0b88146080 unknown :-1 s=CLOSED pgs=4858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 -- 172.21.15.97:0/2571550642 >> 172.21.15.97:0/2571550642 conn(0x7f0b880b41f0 msgr2=0x7f0b880b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 -- 172.21.15.97:0/2571550642 shutdown_connections 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 1 -- 172.21.15.97:0/2571550642 wait complete. 2023-12-07T07:55:50.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 10 monclient: build_initial_monmap 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.714+0000 7f0b8e094640 10 monclient: monmap: 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 1 librados: starting msgr at 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 1 librados: starting objecter 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 1 Processor -- start 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 1 -- start start 2023-12-07T07:55:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 1 librados: setting wanted keys 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 1 librados: calling monclient init 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 10 monclient: init 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.715+0000 7f0b8e094640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0b880b20f0 0x7f0b8813bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): picked mon.noname-c con 0x7f0b880b20f0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b880b2520 0x7f0b8813c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): picked mon.noname-a con 0x7f0b880b2520 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 0x7f0b881dbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): picked mon.noname-b con 0x7f0b88145840 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880b1580 con 0x7f0b880b2520 2023-12-07T07:55:50.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b881dba30 con 0x7f0b88145840 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88071a30 con 0x7f0b880b20f0 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b8e094640 10 monclient(hunting): authenticate will time out at 2673.094407s 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b7effd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b880b2520 0x7f0b8813c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b877fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0b880b20f0 0x7f0b8813bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b87fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 0x7f0b881dbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b7effd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b880b2520 0x7f0b8813c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45814/0 (socket says 172.21.15.97:45814) 2023-12-07T07:55:50.726 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b7effd640 1 -- 172.21.15.97:0/2410418506 learned_addr learned my addr 172.21.15.97:0/2410418506 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b7effd640 10 monclient(hunting): get_auth_request con 0x7f0b880b2520 auth_method 0 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b7effd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b7effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b87fff640 10 monclient(hunting): get_auth_request con 0x7f0b88145840 auth_method 0 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b87fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.716+0000 7f0b87fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b877fe640 10 monclient(hunting): get_auth_request con 0x7f0b880b20f0 auth_method 0 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b877fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b877fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.717+0000 7f0b877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 10 monclient(hunting): handle_auth_done global_id 20194 payload 824 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 1 -- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0b880b20f0 msgr2=0x7f0b8813bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0b880b20f0 0x7f0b8813bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 1 -- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b880b2520 msgr2=0x7f0b8813c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b880b2520 0x7f0b8813c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 1 monclient: found mon.noname-b 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.720+0000 7f0b87fff640 1 -- 172.21.15.97:0/2410418506 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009400 con 0x7f0b88145840 2023-12-07T07:55:50.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.722+0000 7f0b87fff640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 0x7f0b881dbec0 secure :-1 s=READY pgs=4859 cs=0 l=1 rev1=1 crypto rx=0x7f0b80007dc0 tx=0x7f0b8000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.722+0000 7f0b84ff9640 10 client.?.objecter ms_handle_connect 0x7f0b88145840 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.722+0000 7f0b84ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80018070 con 0x7f0b88145840 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b84ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b84ff9640 10 monclient: dump: 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.732 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b84ff9640 10 monclient: _finish_auth 0 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b84ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 5 monclient: authenticate success, global_id 20194 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 10 monclient: _renew_subs 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8806d820 con 0x7f0b88145840 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 10 client.20194.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 10 monclient: _renew_subs 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b881dc5e0 con 0x7f0b88145840 2023-12-07T07:55:50.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.724+0000 7f0b8e094640 1 librados: init done 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b8e094640 10 client.20194.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b8e094640 15 client.20194.objecter _session_command_op_assign -1 1 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b8e094640 20 client.20194.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b8e094640 10 client.20194.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b8e094640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b880aad80 con 0x7f0b88145840 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8000edd0 con 0x7f0b88145840 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b84ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80003b90 con 0x7f0b88145840 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b84ff9640 10 monclient: dump: 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.735 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.725+0000 7f0b867fc640 4 set_mon_vals no callback set 2023-12-07T07:55:50.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.745+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0b80014420 con 0x7f0b88145840 2023-12-07T07:55:50.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.745+0000 7f0b84ff9640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0b6806c7d0 0x7f0b6806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.745+0000 7f0b877fe640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0b6806c7d0 0x7f0b6806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.745+0000 7f0b877fe640 10 monclient: get_auth_request con 0x7f0b6806c7d0 auth_method 0 2023-12-07T07:55:50.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.745+0000 7f0b877fe640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0b6806c7d0 0x7f0b6806ec90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0b880b7600 tx=0x7f0b7803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.745+0000 7f0b84ff9640 10 client.20194.objecter ms_handle_connect 0x7f0b6806c7d0 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f0b800593d0 con 0x7f0b88145840 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 10 client.20194.objecter ms_dispatch 0x7f0b88000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 3 client.20194.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 3 client.20194.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 10 client.20194.objecter checking command 1 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f0b68074100 0x7f0b680765a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 20 client.20194.objecter _get_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 20 client.20194.objecter put_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 15 client.20194.objecter _session_command_op_remove -1 1 2023-12-07T07:55:50.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 20 client.20194.objecter _get_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 20 client.20194.objecter get_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 15 client.20194.objecter _session_command_op_assign 1 1 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 20 client.20194.objecter put_session s=0x7f0b68073a30 osd=1 4 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 10 client.20194.objecter _send_command 1 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 --> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b68076c70 con 0x7f0b68074100 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 20 client.20194.objecter dump_active .. 0 homeless 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b80055710 con 0x7f0b88145840 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.748+0000 7f0b84ff9640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:50.752 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.749+0000 7f0b7effd640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f0b68074100 0x7f0b680765a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.749+0000 7f0b7effd640 10 monclient: get_auth_request con 0x7f0b68074100 auth_method 0 2023-12-07T07:55:50.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.751+0000 7f0b7effd640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f0b68074100 0x7f0b680765a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.751+0000 7f0b84ff9640 10 client.20194.objecter ms_handle_connect 0x7f0b68074100 2023-12-07T07:55:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.774+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== osd.1 v2:172.21.15.97:6826/138440183 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0b68076c70 con 0x7f0b68074100 2023-12-07T07:55:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.774+0000 7f0b84ff9640 10 client.20194.objecter ms_dispatch 0x7f0b88000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.774+0000 7f0b84ff9640 10 client.20194.objecter _finish_command 1 = system:0 2023-12-07T07:55:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.774+0000 7f0b84ff9640 20 client.20194.objecter put_session s=0x7f0b68073a30 osd=1 4 2023-12-07T07:55:50.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.774+0000 7f0b84ff9640 15 client.20194.objecter _session_command_op_remove 1 1 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 10 client.20194.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 15 client.20194.objecter _session_command_op_assign -1 2 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 20 client.20194.objecter _get_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 20 client.20194.objecter put_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 20 client.20194.objecter _get_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 15 client.20194.objecter _session_command_op_remove -1 2 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 20 client.20194.objecter get_session s=0x7f0b68073a30 osd=1 3 2023-12-07T07:55:50.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 15 client.20194.objecter _session_command_op_assign 1 2 2023-12-07T07:55:50.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 20 client.20194.objecter put_session s=0x7f0b68073a30 osd=1 4 2023-12-07T07:55:50.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 10 client.20194.objecter _send_command 2 2023-12-07T07:55:50.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.787+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 --> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0b880aaaa0 con 0x7f0b68074100 2023-12-07T07:55:50.802 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b84ff9640 1 -- 172.21.15.97:0/2410418506 <== osd.1 v2:172.21.15.97:6826/138440183 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0b70004ae0 con 0x7f0b68074100 2023-12-07T07:55:50.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b84ff9640 10 client.20194.objecter ms_dispatch 0x7f0b88000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:50.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b84ff9640 10 client.20194.objecter _finish_command 2 = system:0 2023-12-07T07:55:50.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b84ff9640 20 client.20194.objecter put_session s=0x7f0b68073a30 osd=1 4 2023-12-07T07:55:50.803 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b84ff9640 15 client.20194.objecter _session_command_op_remove 1 2 2023-12-07T07:55:50.803 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:50.803 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:50.803 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 10 librados: watch_flush enter 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 10 librados: watch_flush exit 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 10 client.20194.objecter close_session for osd.1 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f0b68074100 msgr2=0x7f0b680765a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6826/138440183,v1:172.21.15.97:6827/138440183] conn(0x7f0b68074100 0x7f0b680765a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 20 client.20194.objecter put_session s=0x7f0b68073a30 osd=1 1 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 20 client.20194.objecter shutdown clearing up homeless session... 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 10 client.20194.objecter successfully canceled tick 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0b6806c7d0 msgr2=0x7f0b6806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0b6806c7d0 0x7f0b6806ec90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0b880b7600 tx=0x7f0b7803d040 comp rx=0 tx=0).stop 2023-12-07T07:55:50.805 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 10 monclient: shutdown 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 msgr2=0x7f0b881dbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 --2- 172.21.15.97:0/2410418506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b88145840 0x7f0b881dbec0 secure :-1 s=READY pgs=4859 cs=0 l=1 rev1=1 crypto rx=0x7f0b80007dc0 tx=0x7f0b8000e4b0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b87fff640 1 -- 172.21.15.97:0/2410418506 reap_dead start 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 shutdown_connections 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.800+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 >> 172.21.15.97:0/2410418506 conn(0x7f0b880b41f0 msgr2=0x7f0b880b7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.801+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 shutdown_connections 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.801+0000 7f0b8e094640 1 -- 172.21.15.97:0/2410418506 wait complete. 2023-12-07T07:55:50.806 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.801+0000 7f0b8e094640 1 librados: shutdown 2023-12-07T07:55:50.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.910+0000 7f695202b640 10 monclient: get_monmap_and_config 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.910+0000 7f695202b640 10 monclient: build_initial_monmap 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.910+0000 7f695202b640 10 monclient: monmap: 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:50.911601+0000 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:50.911601+0000 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.913 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:50.914 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:50.914 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:50.914 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:50.914 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:50.914 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.911+0000 7f695202b640 1 Processor -- start 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.911+0000 7f695202b640 1 -- start start 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.911+0000 7f695202b640 10 monclient: init 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f694c06f570 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f694c13eb10 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f694c148360 0x7f694c14a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient(hunting): picked mon.noname-a con 0x7f694c148360 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c071550 con 0x7f694c13eb10 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c071830 con 0x7f694c06f570 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f694c071b10 con 0x7f694c148360 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f695202b640 10 monclient(hunting): authenticate will time out at 2673.290517s 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f694bfff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f694c148360 0x7f694c14a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f694b7fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f694c06f570 0x7f694c13bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:53836/0 (socket says 172.21.15.97:53836) 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.912+0000 7f694b7fe640 1 -- 172.21.15.97:0/3876562356 learned_addr learned my addr 172.21.15.97:0/3876562356 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.913+0000 7f694bfff640 10 monclient(hunting): get_auth_request con 0x7f694c148360 auth_method 0 2023-12-07T07:55:50.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.913+0000 7f694bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.913+0000 7f694bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.913+0000 7f694bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.914+0000 7f694bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.914+0000 7f694bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.914+0000 7f694bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694a7fc640 1 -- 172.21.15.97:0/3876562356 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240836953 0 0) 0x7f694c071830 con 0x7f694c06f570 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694a7fc640 10 monclient(hunting): _negotiate 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694a7fc640 10 monclient(hunting): my global_id is 20197 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694a7fc640 1 -- 172.21.15.97:0/3876562356 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6934005730 con 0x7f694c06f570 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 10 monclient(hunting): handle_auth_done global_id 16140 payload 274 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 1 -- 172.21.15.97:0/3876562356 >> v1:172.21.15.154:6789/0 conn(0x7f694c06f570 legacy=0x7f694c13bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 1 -- 172.21.15.97:0/3876562356 >> v1:172.21.15.97:6789/0 conn(0x7f694c13eb10 legacy=0x7f694c13ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 1 monclient: found mon.noname-a 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:50.920 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.916+0000 7f694bfff640 1 -- 172.21.15.97:0/3876562356 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c0aecb0 con 0x7f694c148360 2023-12-07T07:55:50.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.921+0000 7f694bfff640 1 --2- 172.21.15.97:0/3876562356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c14a750 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f6944009d90 tx=0x7f694402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59efcea53611cf31 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f694a7fc640 1 -- 172.21.15.97:0/3876562356 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694402cc60 con 0x7f694c148360 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f694a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f694a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f694a7fc640 10 monclient: dump: 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f694a7fc640 10 monclient: _finish_auth 0 2023-12-07T07:55:50.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f694a7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:55:50.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f694a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:50.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f695202b640 5 monclient: authenticate success, global_id 16140 2023-12-07T07:55:50.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.923+0000 7f695202b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:50.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.924+0000 7f694a7fc640 1 -- 172.21.15.97:0/3876562356 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694402cdc0 con 0x7f694c148360 2023-12-07T07:55:50.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.924+0000 7f694a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:50.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.924+0000 7f695202b640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:50.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.924+0000 7f695202b640 4 set_mon_vals no callback set 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 10 monclient: shutdown 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 1 -- 172.21.15.97:0/3876562356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 msgr2=0x7f694c14a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 1 --2- 172.21.15.97:0/3876562356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c14a750 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f6944009d90 tx=0x7f694402beb0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 1 -- 172.21.15.97:0/3876562356 shutdown_connections 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 1 --2- 172.21.15.97:0/3876562356 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c14a750 unknown :-1 s=CLOSED pgs=3219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 1 -- 172.21.15.97:0/3876562356 >> 172.21.15.97:0/3876562356 conn(0x7f694c06c100 msgr2=0x7f694c0b4660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 1 -- 172.21.15.97:0/3876562356 shutdown_connections 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 1 -- 172.21.15.97:0/3876562356 wait complete. 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 10 monclient: build_initial_monmap 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.925+0000 7f695202b640 10 monclient: monmap: 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:50.929 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.926+0000 7f695202b640 1 librados: starting msgr at 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.926+0000 7f695202b640 1 librados: starting objecter 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.926+0000 7f695202b640 1 Processor -- start 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.926+0000 7f695202b640 1 -- start start 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.926+0000 7f695202b640 1 librados: setting wanted keys 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.926+0000 7f695202b640 1 librados: calling monclient init 2023-12-07T07:55:50.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.926+0000 7f695202b640 10 monclient: init 2023-12-07T07:55:50.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:50.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:55:50.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f694c06f570 0x7f694c0b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): picked mon.noname-c con 0x7f694c06f570 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f694c13eb10 0x7f694c1d60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): picked mon.noname-b con 0x7f694c13eb10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c1d6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): picked mon.noname-a con 0x7f694c148360 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c136e70 con 0x7f694c148360 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c0b37a0 con 0x7f694c13eb10 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c137300 con 0x7f694c06f570 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f695202b640 10 monclient(hunting): authenticate will time out at 2673.305428s 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c1d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f694c13eb10 0x7f694c1d60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f694c06f570 0x7f694c0b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:50.935 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c1d6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45834/0 (socket says 172.21.15.97:45834) 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694bfff640 1 -- 172.21.15.97:0/2130509538 learned_addr learned my addr 172.21.15.97:0/2130509538 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694bfff640 10 monclient(hunting): get_auth_request con 0x7f694c148360 auth_method 0 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694affd640 10 monclient(hunting): get_auth_request con 0x7f694c13eb10 auth_method 0 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.936 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.927+0000 7f694affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694b7fe640 10 monclient(hunting): get_auth_request con 0x7f694c06f570 auth_method 0 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.937 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.928+0000 7f694b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 10 monclient(hunting): handle_auth_done global_id 20200 payload 824 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 1 -- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f694c06f570 msgr2=0x7f694c0b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.938 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f694c06f570 0x7f694c0b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 1 -- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 msgr2=0x7f694c1d6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c1d6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 10 monclient: _finish_hunting 0 2023-12-07T07:55:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 1 monclient: found mon.noname-b 2023-12-07T07:55:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.939 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.931+0000 7f694affd640 1 -- 172.21.15.97:0/2130509538 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944009a70 con 0x7f694c13eb10 2023-12-07T07:55:50.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.933+0000 7f694affd640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f694c13eb10 0x7f694c1d60f0 secure :-1 s=READY pgs=4861 cs=0 l=1 rev1=1 crypto rx=0x7f694000c390 tx=0x7f694000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:50.940 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.933+0000 7f6933fff640 10 client.?.objecter ms_handle_connect 0x7f694c13eb10 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.933+0000 7f6933fff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 1 -- 172.21.15.97:0/2130509538 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400052d0 con 0x7f694c13eb10 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: dump: 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.941 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: _finish_auth 0 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: _check_auth_tickets 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 1 -- 172.21.15.97:0/2130509538 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6940005430 con 0x7f694c13eb10 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 5 monclient: authenticate success, global_id 20200 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 10 monclient: _renew_subs 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 1 -- 172.21.15.97:0/2130509538 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940003830 con 0x7f694c13eb10 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c1374c0 con 0x7f694c13eb10 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6933fff640 10 monclient: dump: 2023-12-07T07:55:50.942 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f6949ffb640 4 set_mon_vals no callback set 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 10 client.20200.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 10 monclient: _renew_subs 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.943 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c136130 con 0x7f694c13eb10 2023-12-07T07:55:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.937+0000 7f695202b640 1 librados: init done 2023-12-07T07:55:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.938+0000 7f695202b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.938+0000 7f695202b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.938+0000 7f695202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:50.938+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694c071f30 con 0x7f694c13eb10 2023-12-07T07:55:51.186 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.183+0000 7f6933fff640 1 -- 172.21.15.97:0/2130509538 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6940003990 con 0x7f694c13eb10 2023-12-07T07:55:51.186 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.183+0000 7f6933fff640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693806c820 0x7f693806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.186 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.183+0000 7f694b7fe640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693806c820 0x7f693806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.186 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.183+0000 7f694b7fe640 10 monclient: get_auth_request con 0x7f693806c820 auth_method 0 2023-12-07T07:55:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.184+0000 7f694b7fe640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693806c820 0x7f693806ece0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f693c005880 tx=0x7f693c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:51.187 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.184+0000 7f6933fff640 10 client.20200.objecter ms_handle_connect 0x7f693806c820 2023-12-07T07:55:51.188 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.185+0000 7f6933fff640 1 -- 172.21.15.97:0/2130509538 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 459..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f6940057d10 con 0x7f694c13eb10 2023-12-07T07:55:51.188 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.185+0000 7f6933fff640 10 client.20200.objecter ms_dispatch 0x7f694c000b90 osd_map(920..920 src has 459..920) v4 2023-12-07T07:55:51.188 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.185+0000 7f6933fff640 3 client.20200.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:51.188 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.185+0000 7f6933fff640 3 client.20200.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:51.188 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.185+0000 7f6933fff640 20 client.20200.objecter dump_active .. 0 homeless 2023-12-07T07:55:51.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.195+0000 7f6933fff640 1 -- 172.21.15.97:0/2130509538 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6940054050 con 0x7f694c13eb10 2023-12-07T07:55:51.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.195+0000 7f6933fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:51.197 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.195+0000 7f6933fff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:51.257 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:17.705 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.263 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:17.712 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.295 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:17.744 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.314 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:17.752 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.318 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:17.756 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.324 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:17.773 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.330 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:17.779 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.348 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:17.785 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.371 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:17.820 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:51.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.390+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f694c06f950 con 0x7f693806c820 2023-12-07T07:55:51.393 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.390+0000 7f6933fff640 1 -- 172.21.15.97:0/2130509538 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f693c001580 con 0x7f693806c820 2023-12-07T07:55:51.393 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 1 to deep-scrub 2023-12-07T07:55:51.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 10 librados: watch_flush enter 2023-12-07T07:55:51.396 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 10 librados: watch_flush exit 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 20 client.20200.objecter shutdown clearing up homeless session... 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 10 client.20200.objecter successfully canceled tick 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693806c820 msgr2=0x7f693806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693806c820 0x7f693806ece0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f693c005880 tx=0x7f693c009040 comp rx=0 tx=0).stop 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 10 monclient: shutdown 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f694c13eb10 msgr2=0x7f694c1d60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f694c13eb10 0x7f694c1d60f0 secure :-1 s=READY pgs=4861 cs=0 l=1 rev1=1 crypto rx=0x7f694000c390 tx=0x7f694000c850 comp rx=0 tx=0).stop 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 shutdown_connections 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693806c820 0x7f693806ece0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f694c148360 0x7f694c1d6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.397 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f694c13eb10 0x7f694c1d60f0 unknown :-1 s=CLOSED pgs=4861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 --2- 172.21.15.97:0/2130509538 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f694c06f570 0x7f694c0b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 >> 172.21.15.97:0/2130509538 conn(0x7f694c06c100 msgr2=0x7f694c06a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 shutdown_connections 2023-12-07T07:55:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.393+0000 7f695202b640 1 -- 172.21.15.97:0/2130509538 wait complete. 2023-12-07T07:55:51.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.394+0000 7f695202b640 1 librados: shutdown 2023-12-07T07:55:51.411 INFO:tasks.ceph:Scrubbing osd.2 2023-12-07T07:55:51.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.518+0000 7f2659b33640 10 monclient: get_monmap_and_config 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.518+0000 7f2659b33640 10 monclient: build_initial_monmap 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.518+0000 7f2659b33640 10 monclient: monmap: 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:51.519713+0000 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:51.519713+0000 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.521 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:51.522 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:51.522 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:51.522 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:51.522 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:51.522 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.519+0000 7f2659b33640 1 Processor -- start 2023-12-07T07:55:51.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.519+0000 7f2659b33640 1 -- start start 2023-12-07T07:55:51.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.519+0000 7f2659b33640 10 monclient: init 2023-12-07T07:55:51.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.519+0000 7f2659b33640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:51.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-12-07T07:55:51.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26540af360 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2654138ba0 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2654142600 0x7f265414a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 10 monclient(hunting): picked mon.noname-a con 0x7f2654142600 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26540720b0 con 0x7f26540af360 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2654070be0 con 0x7f2654138ba0 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2654072320 con 0x7f2654142600 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:51.524 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.520+0000 7f2659b33640 10 monclient(hunting): authenticate will time out at 2673.898394s 2023-12-07T07:55:51.537 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.535+0000 7f2653fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2654142600 0x7f265414a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.535+0000 7f264affd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f2654138ba0 0x7f2654141ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:53838/0 (socket says 172.21.15.97:53838) 2023-12-07T07:55:51.538 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.535+0000 7f264affd640 1 -- 172.21.15.97:0/4174580505 learned_addr learned my addr 172.21.15.97:0/4174580505 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:51.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.555+0000 7f2653fff640 10 monclient(hunting): get_auth_request con 0x7f2654142600 auth_method 0 2023-12-07T07:55:51.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.555+0000 7f2653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:51.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.555+0000 7f2653fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:51.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.555+0000 7f2653fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:51.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.564+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:51.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.564+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:51.566 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.564+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 10 monclient(hunting): handle_auth_done global_id 16152 payload 274 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 1 -- 172.21.15.97:0/4174580505 >> v1:172.21.15.154:6789/0 conn(0x7f2654138ba0 legacy=0x7f2654141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 1 -- 172.21.15.97:0/4174580505 >> v1:172.21.15.97:6789/0 conn(0x7f26540af360 legacy=0x7f2654138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 1 monclient: found mon.noname-a 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:51.569 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.566+0000 7f2653fff640 1 -- 172.21.15.97:0/4174580505 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540afde0 con 0x7f2654142600 2023-12-07T07:55:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.574+0000 7f2653fff640 1 --2- 172.21.15.97:0/4174580505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f265414a290 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f264c009d90 tx=0x7f264c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bea4dde57ba230 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 1 -- 172.21.15.97:0/4174580505 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c02cc60 con 0x7f2654142600 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: dump: 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.613 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: _finish_auth 0 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 1 -- 172.21.15.97:0/4174580505 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c02cdc0 con 0x7f2654142600 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2659b33640 5 monclient: authenticate success, global_id 16152 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2659b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2659b33640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 1 -- 172.21.15.97:0/4174580505 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c034660 con 0x7f2654142600 2023-12-07T07:55:51.614 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2659b33640 4 set_mon_vals no callback set 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.610+0000 7f2652ffd640 10 monclient: dump: 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.615 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 10 monclient: shutdown 2023-12-07T07:55:51.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:51.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 1 -- 172.21.15.97:0/4174580505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 msgr2=0x7f265414a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 1 --2- 172.21.15.97:0/4174580505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f265414a290 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f264c009d90 tx=0x7f264c02beb0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.617 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 1 -- 172.21.15.97:0/4174580505 shutdown_connections 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 1 --2- 172.21.15.97:0/4174580505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f265414a290 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 1 -- 172.21.15.97:0/4174580505 >> 172.21.15.97:0/4174580505 conn(0x7f26540b01e0 msgr2=0x7f26540b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 1 -- 172.21.15.97:0/4174580505 shutdown_connections 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.611+0000 7f2659b33640 1 -- 172.21.15.97:0/4174580505 wait complete. 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 10 monclient: build_initial_monmap 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 10 monclient: monmap: 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 1 librados: starting msgr at 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 1 librados: starting objecter 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 1 Processor -- start 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 1 -- start start 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 1 librados: setting wanted keys 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 1 librados: calling monclient init 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.612+0000 7f2659b33640 10 monclient: init 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26540af360 0x7f26541cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): picked mon.noname-b con 0x7f26540af360 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2654138ba0 0x7f2654140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): picked mon.noname-c con 0x7f2654138ba0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:51.619 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f2654134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): picked mon.noname-a con 0x7f2654142600 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26541410b0 con 0x7f2654142600 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2654140d90 con 0x7f26540af360 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265413fdb0 con 0x7f2654138ba0 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.613+0000 7f2659b33640 10 monclient(hunting): authenticate will time out at 2673.991838s 2023-12-07T07:55:51.620 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.618+0000 7f2653fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f2654134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.621 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.619+0000 7f264affd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2654138ba0 0x7f2654140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.622 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.620+0000 7f2653fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f2654134bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45852/0 (socket says 172.21.15.97:45852) 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.620+0000 7f2653fff640 1 -- 172.21.15.97:0/473288794 learned_addr learned my addr 172.21.15.97:0/473288794 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.620+0000 7f2653fff640 10 monclient(hunting): get_auth_request con 0x7f2654142600 auth_method 0 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.620+0000 7f2653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.620+0000 7f2653fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.620+0000 7f2653fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.621+0000 7f264affd640 10 monclient(hunting): get_auth_request con 0x7f2654138ba0 auth_method 0 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.621+0000 7f264affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:51.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.621+0000 7f264affd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:51.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.621+0000 7f264affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.623+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.623+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.623+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.623+0000 7f264affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.623+0000 7f264affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:51.626 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.623+0000 7f264affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:51.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 10 monclient(hunting): handle_auth_done global_id 16155 payload 824 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 1 -- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2654138ba0 msgr2=0x7f2654140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2654138ba0 0x7f2654140850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 1 -- 172.21.15.97:0/473288794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26540af360 msgr2=0x7f26541cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26540af360 0x7f26541cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 1 monclient: found mon.noname-a 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:51.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.628+0000 7f2653fff640 1 -- 172.21.15.97:0/473288794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c009a70 con 0x7f2654142600 2023-12-07T07:55:51.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.659+0000 7f2653fff640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f2654134bd0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f264c02c4c0 tx=0x7f264c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:51.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.659+0000 7f2650ff9640 10 client.?.objecter ms_handle_connect 0x7f2654142600 2023-12-07T07:55:51.662 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.659+0000 7f2650ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c03e610 con 0x7f2654142600 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: dump: 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.673 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: _finish_auth 0 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c03e770 con 0x7f2654142600 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c03eb20 con 0x7f2654142600 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 5 monclient: authenticate success, global_id 16155 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 10 monclient: _renew_subs 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:51.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265413ffd0 con 0x7f2654142600 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f26527fc640 4 set_mon_vals no callback set 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2650ff9640 10 monclient: dump: 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.675 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 10 client.16155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 10 monclient: _renew_subs 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26541352f0 con 0x7f2654142600 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.670+0000 7f2659b33640 1 librados: init done 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.671+0000 7f2659b33640 10 client.16155.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:51.676 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.671+0000 7f2659b33640 15 client.16155.objecter _session_command_op_assign -1 1 2023-12-07T07:55:51.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.671+0000 7f2659b33640 20 client.16155.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:51.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.671+0000 7f2659b33640 10 client.16155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:51.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.671+0000 7f2659b33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:51.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.671+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f265406e7c0 con 0x7f2654142600 2023-12-07T07:55:51.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.707+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f264c03b050 con 0x7f2654142600 2023-12-07T07:55:51.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.708+0000 7f2650ff9640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f263406c820 0x7f263406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.708+0000 7f26537fe640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f263406c820 0x7f263406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.708+0000 7f26537fe640 10 monclient: get_auth_request con 0x7f263406c820 auth_method 0 2023-12-07T07:55:51.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.708+0000 7f26537fe640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f263406c820 0x7f263406ece0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2644000f60 tx=0x7f26440079d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:51.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.708+0000 7f2650ff9640 10 client.16155.objecter ms_handle_connect 0x7f263406c820 2023-12-07T07:55:51.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f264c07b160 con 0x7f2654142600 2023-12-07T07:55:51.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 10 client.16155.objecter ms_dispatch 0x7f2654000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:51.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 3 client.16155.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:51.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 3 client.16155.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:51.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 10 client.16155.objecter checking command 1 2023-12-07T07:55:51.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7f2634074150 0x7f26340765f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 20 client.16155.objecter _get_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 20 client.16155.objecter put_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 15 client.16155.objecter _session_command_op_remove -1 1 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 20 client.16155.objecter _get_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 20 client.16155.objecter get_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 15 client.16155.objecter _session_command_op_assign 2 1 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 20 client.16155.objecter put_session s=0x7f2634073a80 osd=2 4 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 10 client.16155.objecter _send_command 1 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 --> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2634076cc0 con 0x7f2634074150 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 20 client.16155.objecter dump_active .. 0 homeless 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7f264c0774a0 con 0x7f2654142600 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.724+0000 7f2650ff9640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:51.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.725+0000 7f264affd640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7f2634074150 0x7f26340765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.725+0000 7f264affd640 10 monclient: get_auth_request con 0x7f2634074150 auth_method 0 2023-12-07T07:55:51.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.728+0000 7f2650ff9640 10 client.16155.objecter ms_handle_connect 0x7f2634074150 2023-12-07T07:55:51.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.728+0000 7f264affd640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7f2634074150 0x7f26340765f0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:51.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.752+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== osd.2 v2:172.21.15.97:6802/1673674356 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2634076cc0 con 0x7f2634074150 2023-12-07T07:55:51.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.752+0000 7f2650ff9640 10 client.16155.objecter ms_dispatch 0x7f2654000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:51.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.752+0000 7f2650ff9640 10 client.16155.objecter _finish_command 1 = system:0 2023-12-07T07:55:51.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.752+0000 7f2650ff9640 20 client.16155.objecter put_session s=0x7f2634073a80 osd=2 4 2023-12-07T07:55:51.754 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.752+0000 7f2650ff9640 15 client.16155.objecter _session_command_op_remove 2 1 2023-12-07T07:55:51.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 10 client.16155.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:51.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 15 client.16155.objecter _session_command_op_assign -1 2 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 20 client.16155.objecter _get_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 20 client.16155.objecter put_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 20 client.16155.objecter _get_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 15 client.16155.objecter _session_command_op_remove -1 2 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 20 client.16155.objecter get_session s=0x7f2634073a80 osd=2 3 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 15 client.16155.objecter _session_command_op_assign 2 2 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 20 client.16155.objecter put_session s=0x7f2634073a80 osd=2 4 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 10 client.16155.objecter _send_command 2 2023-12-07T07:55:51.768 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.765+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 --> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2654135720 con 0x7f2634074150 2023-12-07T07:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.778+0000 7f2650ff9640 1 -- 172.21.15.97:0/473288794 <== osd.2 v2:172.21.15.97:6802/1673674356 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2654135720 con 0x7f2634074150 2023-12-07T07:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.778+0000 7f2650ff9640 10 client.16155.objecter ms_dispatch 0x7f2654000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.778+0000 7f2650ff9640 10 client.16155.objecter _finish_command 2 = system:0 2023-12-07T07:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.778+0000 7f2650ff9640 20 client.16155.objecter put_session s=0x7f2634073a80 osd=2 4 2023-12-07T07:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.778+0000 7f2650ff9640 15 client.16155.objecter _session_command_op_remove 2 2 2023-12-07T07:55:51.781 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:51.782 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:51.782 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 10 librados: watch_flush enter 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 10 librados: watch_flush exit 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 10 client.16155.objecter close_session for osd.2 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7f2634074150 msgr2=0x7f26340765f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6802/1673674356,v1:172.21.15.97:6803/1673674356] conn(0x7f2634074150 0x7f26340765f0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 20 client.16155.objecter put_session s=0x7f2634073a80 osd=2 1 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 20 client.16155.objecter shutdown clearing up homeless session... 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 10 client.16155.objecter successfully canceled tick 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f263406c820 msgr2=0x7f263406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f263406c820 0x7f263406ece0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2644000f60 tx=0x7f26440079d0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 10 monclient: shutdown 2023-12-07T07:55:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 msgr2=0x7f2654134bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 --2- 172.21.15.97:0/473288794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2654142600 0x7f2654134bd0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f264c02c4c0 tx=0x7f264c004a40 comp rx=0 tx=0).stop 2023-12-07T07:55:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2653fff640 1 -- 172.21.15.97:0/473288794 reap_dead start 2023-12-07T07:55:51.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 shutdown_connections 2023-12-07T07:55:51.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 >> 172.21.15.97:0/473288794 conn(0x7f26540b01e0 msgr2=0x7f265406ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:51.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 shutdown_connections 2023-12-07T07:55:51.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 -- 172.21.15.97:0/473288794 wait complete. 2023-12-07T07:55:51.785 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.779+0000 7f2659b33640 1 librados: shutdown 2023-12-07T07:55:51.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.893+0000 7fc30c961640 10 monclient: get_monmap_and_config 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.893+0000 7fc30c961640 10 monclient: build_initial_monmap 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.893+0000 7fc30c961640 10 monclient: monmap: 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:51.894978+0000 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:51.894978+0000 2023-12-07T07:55:51.896 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:51.897 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.897 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:51.897 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:51.897 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:51.897 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:51.897 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:51.897 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.894+0000 7fc30c961640 1 Processor -- start 2023-12-07T07:55:51.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.894+0000 7fc30c961640 1 -- start start 2023-12-07T07:55:51.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.894+0000 7fc30c961640 10 monclient: init 2023-12-07T07:55:51.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:51.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-07T07:55:51.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3080a8910 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:51.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3080a9b70 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc30813f370 0x7fc3081426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient(hunting): picked mon.noname-c con 0x7fc30813f370 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3080a8cf0 con 0x7fc3080a9b70 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3080aabd0 con 0x7fc3080a8910 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080aaeb0 con 0x7fc30813f370 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:51.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.895+0000 7fc30c961640 10 monclient(hunting): authenticate will time out at 2674.273707s 2023-12-07T07:55:51.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.899+0000 7fc305d9b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc3080a9b70 0x7fc308141f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47062/0 (socket says 172.21.15.97:47062) 2023-12-07T07:55:51.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.899+0000 7fc305d9b640 1 -- 172.21.15.97:0/1919163202 learned_addr learned my addr 172.21.15.97:0/1919163202 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:51.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.908+0000 7fc306d9d640 1 --2- 172.21.15.97:0/1919163202 >> v2:172.21.15.154:3300/0 conn(0x7fc30813f370 0x7fc3081426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.910+0000 7fc306d9d640 10 monclient(hunting): get_auth_request con 0x7fc30813f370 auth_method 0 2023-12-07T07:55:51.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.910+0000 7fc306d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:51.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.910+0000 7fc306d9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:51.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.910+0000 7fc306d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:51.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.912+0000 7fc306d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:51.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.912+0000 7fc306d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:51.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.912+0000 7fc306d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:51.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.914+0000 7fc30559a640 1 -- 172.21.15.97:0/1919163202 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307482332 0 0) 0x7fc3080a8cf0 con 0x7fc3080a9b70 2023-12-07T07:55:51.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.914+0000 7fc30559a640 10 monclient(hunting): _negotiate 2023-12-07T07:55:51.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.914+0000 7fc30559a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:51.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.914+0000 7fc30559a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:51.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.914+0000 7fc30559a640 10 monclient(hunting): my global_id is 16158 2023-12-07T07:55:51.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.914+0000 7fc30559a640 1 -- 172.21.15.97:0/1919163202 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0003520 con 0x7fc3080a9b70 2023-12-07T07:55:51.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 10 monclient(hunting): handle_auth_done global_id 20206 payload 274 2023-12-07T07:55:51.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 1 -- 172.21.15.97:0/1919163202 >> v1:172.21.15.154:6789/0 conn(0x7fc3080a8910 legacy=0x7fc30813ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 1 -- 172.21.15.97:0/1919163202 >> v1:172.21.15.97:6789/0 conn(0x7fc3080a9b70 legacy=0x7fc308141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 10 monclient: _finish_hunting 0 2023-12-07T07:55:51.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 1 monclient: found mon.noname-c 2023-12-07T07:55:51.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:51.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:51.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.923+0000 7fc306d9d640 1 -- 172.21.15.97:0/1919163202 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3080a9720 con 0x7fc30813f370 2023-12-07T07:55:51.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.945+0000 7fc306d9d640 1 --2- 172.21.15.97:0/1919163202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30813f370 0x7fc3081426a0 secure :-1 s=READY pgs=4864 cs=0 l=1 rev1=1 crypto rx=0x7fc300009d90 tx=0x7fc30002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde1d10a7142a317 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:51.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30559a640 1 -- 172.21.15.97:0/1919163202 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30002cd00 con 0x7fc30813f370 2023-12-07T07:55:51.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:51.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30559a640 10 monclient: dump: 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30559a640 10 monclient: _finish_auth 0 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30559a640 10 monclient: _check_auth_tickets 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30c961640 5 monclient: authenticate success, global_id 20206 2023-12-07T07:55:51.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.957+0000 7fc30c961640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30559a640 1 -- 172.21.15.97:0/1919163202 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30002ce60 con 0x7fc30813f370 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30559a640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30559a640 1 -- 172.21.15.97:0/1919163202 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300034640 con 0x7fc30813f370 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30559a640 10 monclient: dump: 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.967 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.968 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.968 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.968 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30c961640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:51.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.964+0000 7fc30c961640 4 set_mon_vals no callback set 2023-12-07T07:55:51.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 10 monclient: shutdown 2023-12-07T07:55:51.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:51.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 1 -- 172.21.15.97:0/1919163202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30813f370 msgr2=0x7fc3081426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 1 --2- 172.21.15.97:0/1919163202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30813f370 0x7fc3081426a0 secure :-1 s=READY pgs=4864 cs=0 l=1 rev1=1 crypto rx=0x7fc300009d90 tx=0x7fc30002bf50 comp rx=0 tx=0).stop 2023-12-07T07:55:51.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 1 -- 172.21.15.97:0/1919163202 shutdown_connections 2023-12-07T07:55:51.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 1 --2- 172.21.15.97:0/1919163202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30813f370 0x7fc3081426a0 unknown :-1 s=CLOSED pgs=4864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 1 -- 172.21.15.97:0/1919163202 >> 172.21.15.97:0/1919163202 conn(0x7fc3080b7780 msgr2=0x7fc3080b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 1 -- 172.21.15.97:0/1919163202 shutdown_connections 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.965+0000 7fc30c961640 1 -- 172.21.15.97:0/1919163202 wait complete. 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.966+0000 7fc30c961640 10 monclient: build_initial_monmap 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.966+0000 7fc30c961640 10 monclient: monmap: 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:51.971 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.966+0000 7fc30c961640 1 librados: starting msgr at 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.966+0000 7fc30c961640 1 librados: starting objecter 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 Processor -- start 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 -- start start 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 librados: setting wanted keys 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 librados: calling monclient init 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient: init 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3080a8910 0x7fc30813aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): picked mon.noname-b con 0x7fc3080a8910 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc30813f370 0x7fc3080ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): picked mon.noname-c con 0x7fc30813f370 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc30813a2b0 0x7fc308132490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): picked mon.noname-a con 0x7fc30813a2b0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080abaf0 con 0x7fc30813a2b0 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080ac960 con 0x7fc3080a8910 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080abd20 con 0x7fc30813f370 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.967+0000 7fc30c961640 10 monclient(hunting): authenticate will time out at 2674.345964s 2023-12-07T07:55:51.973 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.970+0000 7fc305d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc30813f370 0x7fc3080ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.972+0000 7fc305d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc30813f370 0x7fc3080ac420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38736/0 (socket says 172.21.15.97:38736) 2023-12-07T07:55:51.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.972+0000 7fc305d9b640 1 -- 172.21.15.97:0/2224811132 learned_addr learned my addr 172.21.15.97:0/2224811132 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:51.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.972+0000 7fc305d9b640 10 monclient(hunting): get_auth_request con 0x7fc30813f370 auth_method 0 2023-12-07T07:55:51.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.972+0000 7fc305d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:51.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.972+0000 7fc305d9b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:51.975 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.972+0000 7fc305d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:51.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc305d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:51.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc305d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:51.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc305d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:51.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc30659c640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3080a8910 0x7fc30813aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:51.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc30659c640 10 monclient(hunting): get_auth_request con 0x7fc3080a8910 auth_method 0 2023-12-07T07:55:51.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc30659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:51.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc30659c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:51.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.974+0000 7fc30659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:51.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.976+0000 7fc30659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:51.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.976+0000 7fc30659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:51.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.976+0000 7fc30659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:51.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 10 monclient(hunting): handle_auth_done global_id 17678 payload 824 2023-12-07T07:55:51.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 1 -- 172.21.15.97:0/2224811132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3080a8910 msgr2=0x7fc30813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3080a8910 0x7fc30813aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 1 -- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc30813a2b0 msgr2=0x7fc308132490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:51.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc30813a2b0 0x7fc308132490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:51.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 10 monclient: _finish_hunting 0 2023-12-07T07:55:51.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 1 monclient: found mon.noname-c 2023-12-07T07:55:51.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:51.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:55:51.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.978+0000 7fc305d9b640 1 -- 172.21.15.97:0/2224811132 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc300009a70 con 0x7fc30813f370 2023-12-07T07:55:51.984 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.982+0000 7fc305d9b640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc30813f370 0x7fc3080ac420 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c500 tx=0x7fc2fc00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:51.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.982+0000 7fc2f6ffd640 10 client.?.objecter ms_handle_connect 0x7fc30813f370 2023-12-07T07:55:51.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.982+0000 7fc2f6ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:51.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 1 -- 172.21.15.97:0/2224811132 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc005ce0 con 0x7fc30813f370 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: dump: 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: _finish_auth 0 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:51.994 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 1 -- 172.21.15.97:0/2224811132 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc005e40 con 0x7fc30813f370 2023-12-07T07:55:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 1 -- 172.21.15.97:0/2224811132 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc007330 con 0x7fc30813f370 2023-12-07T07:55:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 5 monclient: authenticate success, global_id 17678 2023-12-07T07:55:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 10 monclient: _renew_subs 2023-12-07T07:55:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3080ac000 con 0x7fc30813f370 2023-12-07T07:55:51.995 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc304d99640 4 set_mon_vals no callback set 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc2f6ffd640 10 monclient: dump: 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:51.997 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:51.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 10 client.17678.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:51.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 10 monclient: _renew_subs 2023-12-07T07:55:51.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:51.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3081329d0 con 0x7fc30813f370 2023-12-07T07:55:51.997 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.989+0000 7fc30c961640 1 librados: init done 2023-12-07T07:55:51.998 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.990+0000 7fc30c961640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:51.998 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.990+0000 7fc30c961640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:51.998 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.990+0000 7fc30c961640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:51.998 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:51.990+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc308142b60 con 0x7fc30813f370 2023-12-07T07:55:52.010 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:18.448 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.012 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:18.449 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.019 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:18.457 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.102 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:18.550 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.103 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:18.552 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.111 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:18.560 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.236+0000 7fc2f6ffd640 1 -- 172.21.15.97:0/2224811132 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc2fc007550 con 0x7fc30813f370 2023-12-07T07:55:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.237+0000 7fc2f6ffd640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2e806c7d0 0x7fc2e806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.239 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.237+0000 7fc30659c640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2e806c7d0 0x7fc2e806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.237+0000 7fc30659c640 10 monclient: get_auth_request con 0x7fc2e806c7d0 auth_method 0 2023-12-07T07:55:52.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.237+0000 7fc2f6ffd640 10 client.17678.objecter ms_handle_connect 0x7fc2e806c7d0 2023-12-07T07:55:52.240 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.237+0000 7fc30659c640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2e806c7d0 0x7fc2e806ec90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8000f60 tx=0x7fc2f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.252 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 1 -- 172.21.15.97:0/2224811132 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fc2fc057f80 con 0x7fc30813f370 2023-12-07T07:55:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 10 client.17678.objecter ms_dispatch 0x7fc308000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 3 client.17678.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 3 client.17678.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 20 client.17678.objecter dump_active .. 0 homeless 2023-12-07T07:55:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 1 -- 172.21.15.97:0/2224811132 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2fc0542c0 con 0x7fc30813f370 2023-12-07T07:55:52.253 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:52.254 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.249+0000 7fc2f6ffd640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:52.327 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:18.776 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.328 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:18.777 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.336 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:18.785 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:52.454 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.451+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc308075b60 con 0x7fc2e806c7d0 2023-12-07T07:55:52.455 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.452+0000 7fc2f6ffd640 1 -- 172.21.15.97:0/2224811132 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc308075b60 con 0x7fc2e806c7d0 2023-12-07T07:55:52.455 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 2 to deep-scrub 2023-12-07T07:55:52.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 10 librados: watch_flush enter 2023-12-07T07:55:52.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 10 librados: watch_flush exit 2023-12-07T07:55:52.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 20 client.17678.objecter shutdown clearing up homeless session... 2023-12-07T07:55:52.458 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 10 client.17678.objecter successfully canceled tick 2023-12-07T07:55:52.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2e806c7d0 msgr2=0x7fc2e806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2e806c7d0 0x7fc2e806ec90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8000f60 tx=0x7fc2f8009040 comp rx=0 tx=0).stop 2023-12-07T07:55:52.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 10 monclient: shutdown 2023-12-07T07:55:52.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:52.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc30813f370 msgr2=0x7fc3080ac420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.459 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc30813f370 0x7fc3080ac420 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00c500 tx=0x7fc2fc00c9c0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 shutdown_connections 2023-12-07T07:55:52.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fc2e806c7d0 0x7fc2e806ec90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc30813a2b0 0x7fc308132490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc30813f370 0x7fc3080ac420 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 --2- 172.21.15.97:0/2224811132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3080a8910 0x7fc30813aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 >> 172.21.15.97:0/2224811132 conn(0x7fc3080b7780 msgr2=0x7fc30813c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:52.460 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 shutdown_connections 2023-12-07T07:55:52.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 -- 172.21.15.97:0/2224811132 wait complete. 2023-12-07T07:55:52.461 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.454+0000 7fc30c961640 1 librados: shutdown 2023-12-07T07:55:52.469 INFO:tasks.ceph:Scrubbing osd.3 2023-12-07T07:55:52.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:52.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.591+0000 7f2ae0bf0640 10 monclient: get_monmap_and_config 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.591+0000 7f2ae0bf0640 10 monclient: build_initial_monmap 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.591+0000 7f2ae0bf0640 10 monclient: monmap: 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:52.592485+0000 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:52.592485+0000 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:52.595 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 1 Processor -- start 2023-12-07T07:55:52.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 1 -- start start 2023-12-07T07:55:52.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient: init 2023-12-07T07:55:52.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:52.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2adc145840 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2adc146080 0x7f2adc13f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient(hunting): picked mon.noname-a con 0x7f2adc146080 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2adc13fa50 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2adc0aa850 con 0x7f2adc145840 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2adc0a9260 con 0x7f2adc13fa50 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2adc145c20 con 0x7f2adc146080 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.592+0000 7f2ae0bf0640 10 monclient(hunting): authenticate will time out at 2674.971219s 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.594+0000 7f2adad9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f2adc13fa50 0x7f2adc14c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:53856/0 (socket says 172.21.15.97:53856) 2023-12-07T07:55:52.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.594+0000 7f2adad9d640 1 -- 172.21.15.97:0/2882394540 learned_addr learned my addr 172.21.15.97:0/2882394540 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:52.598 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.596+0000 7f2ad3fff640 1 --2- 172.21.15.97:0/2882394540 >> v2:172.21.15.97:3300/0 conn(0x7f2adc146080 0x7f2adc13f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.597+0000 7f2ad3fff640 10 monclient(hunting): get_auth_request con 0x7f2adc146080 auth_method 0 2023-12-07T07:55:52.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.597+0000 7f2ad3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:52.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.597+0000 7f2ad3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.599 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.597+0000 7f2ad3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:52.600 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.598+0000 7f2ad3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:52.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.598+0000 7f2ad3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:52.601 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.598+0000 7f2ad3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 10 monclient(hunting): handle_auth_done global_id 16161 payload 274 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 1 -- 172.21.15.97:0/2882394540 >> v1:172.21.15.154:6789/0 conn(0x7f2adc13fa50 legacy=0x7f2adc14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 1 -- 172.21.15.97:0/2882394540 >> v1:172.21.15.97:6789/0 conn(0x7f2adc145840 legacy=0x7f2adc0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 1 monclient: found mon.noname-a 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:52.603 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.600+0000 7f2ad3fff640 1 -- 172.21.15.97:0/2882394540 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc0aaaa0 con 0x7f2adc146080 2023-12-07T07:55:52.613 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.610+0000 7f2ad3fff640 1 --2- 172.21.15.97:0/2882394540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc13f510 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40097a0 tx=0x7f2ac402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecfc21a9861f3cb8 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.622 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 1 -- 172.21.15.97:0/2882394540 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac402e070 con 0x7f2adc146080 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: dump: 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.623 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: _finish_auth 0 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: _check_auth_tickets 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 1 -- 172.21.15.97:0/2882394540 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac40315a0 con 0x7f2adc146080 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ae0bf0640 5 monclient: authenticate success, global_id 16161 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ae0bf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ae0bf0640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ae0bf0640 4 set_mon_vals no callback set 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 1 -- 172.21.15.97:0/2882394540 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4031930 con 0x7f2adc146080 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.620+0000 7f2ad9d9b640 10 monclient: dump: 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.625 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 10 monclient: shutdown 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/2882394540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 msgr2=0x7f2adc13f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 1 --2- 172.21.15.97:0/2882394540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc13f510 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f2ac40097a0 tx=0x7f2ac402b840 comp rx=0 tx=0).stop 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/2882394540 shutdown_connections 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 1 --2- 172.21.15.97:0/2882394540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc13f510 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/2882394540 >> 172.21.15.97:0/2882394540 conn(0x7f2adc0b41f0 msgr2=0x7f2adc0b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/2882394540 shutdown_connections 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.621+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/2882394540 wait complete. 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 10 monclient: build_initial_monmap 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 10 monclient: monmap: 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:52.628 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 1 librados: starting msgr at 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 1 librados: starting objecter 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 1 Processor -- start 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 1 -- start start 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 1 librados: setting wanted keys 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 1 librados: calling monclient init 2023-12-07T07:55:52.629 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.622+0000 7f2ae0bf0640 10 monclient: init 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2adc13fa50 0x7f2adc13a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): picked mon.noname-c con 0x7f2adc13fa50 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc145840 0x7f2adc1cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): picked mon.noname-b con 0x7f2adc145840 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc141910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): picked mon.noname-a con 0x7f2adc146080 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0ac130 con 0x7f2adc146080 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0ac410 con 0x7f2adc145840 2023-12-07T07:55:52.630 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc13fe50 con 0x7f2adc13fa50 2023-12-07T07:55:52.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:52.631 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.623+0000 7f2ae0bf0640 10 monclient(hunting): authenticate will time out at 2675.001456s 2023-12-07T07:55:52.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.630+0000 7f2adad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc141910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.631+0000 7f2adad9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc141910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45876/0 (socket says 172.21.15.97:45876) 2023-12-07T07:55:52.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.631+0000 7f2adad9d640 1 -- 172.21.15.97:0/1242735785 learned_addr learned my addr 172.21.15.97:0/1242735785 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:52.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.631+0000 7f2adad9d640 10 monclient(hunting): get_auth_request con 0x7f2adc146080 auth_method 0 2023-12-07T07:55:52.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.631+0000 7f2adad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:52.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.631+0000 7f2adad9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.631+0000 7f2adad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:52.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.636+0000 7f2adad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:52.638 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.636+0000 7f2adad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:52.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.636+0000 7f2adad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:52.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 10 monclient(hunting): handle_auth_done global_id 16164 payload 824 2023-12-07T07:55:52.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 1 -- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2adc13fa50 msgr2=0x7f2adc13a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2adc13fa50 0x7f2adc13a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 1 -- 172.21.15.97:0/1242735785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc145840 msgr2=0x7f2adc1cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.646 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc145840 0x7f2adc1cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 10 monclient: _finish_hunting 0 2023-12-07T07:55:52.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 1 monclient: found mon.noname-a 2023-12-07T07:55:52.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:52.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:52.647 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.643+0000 7f2adad9d640 1 -- 172.21.15.97:0/1242735785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4009400 con 0x7f2adc146080 2023-12-07T07:55:52.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.663+0000 7f2adad9d640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc141910 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400b970 tx=0x7f2ad400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.666 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.663+0000 7f2ad2ffd640 10 client.?.objecter ms_handle_connect 0x7f2adc146080 2023-12-07T07:55:52.667 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.663+0000 7f2ad2ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad400cb40 con 0x7f2adc146080 2023-12-07T07:55:52.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:52.675 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: dump: 2023-12-07T07:55:52.675 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.676 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.677 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: _finish_auth 0 2023-12-07T07:55:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:52.677 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad400cca0 con 0x7f2adc146080 2023-12-07T07:55:52.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 5 monclient: authenticate success, global_id 16164 2023-12-07T07:55:52.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 10 monclient: _renew_subs 2023-12-07T07:55:52.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:52.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2adc140130 con 0x7f2adc146080 2023-12-07T07:55:52.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:52.678 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4014680 con 0x7f2adc146080 2023-12-07T07:55:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad2ffd640 10 monclient: dump: 2023-12-07T07:55:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.679 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.680 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.680 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ad959a640 4 set_mon_vals no callback set 2023-12-07T07:55:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 10 client.16164.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 10 monclient: _renew_subs 2023-12-07T07:55:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2adc140830 con 0x7f2adc146080 2023-12-07T07:55:52.681 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.670+0000 7f2ae0bf0640 1 librados: init done 2023-12-07T07:55:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.671+0000 7f2ae0bf0640 10 client.16164.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.671+0000 7f2ae0bf0640 15 client.16164.objecter _session_command_op_assign -1 1 2023-12-07T07:55:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.671+0000 7f2ae0bf0640 20 client.16164.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.671+0000 7f2ae0bf0640 10 client.16164.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.671+0000 7f2ae0bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:52.682 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.671+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2adc1464b0 con 0x7f2adc146080 2023-12-07T07:55:52.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.705+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2ad401d3b0 con 0x7f2adc146080 2023-12-07T07:55:52.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.705+0000 7f2ad2ffd640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2abc06c820 0x7f2abc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.705+0000 7f2ada59c640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2abc06c820 0x7f2abc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.705+0000 7f2ada59c640 10 monclient: get_auth_request con 0x7f2abc06c820 auth_method 0 2023-12-07T07:55:52.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.706+0000 7f2ada59c640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2abc06c820 0x7f2abc06ece0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2acc000f60 tx=0x7f2acc0079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.706+0000 7f2ad2ffd640 10 client.16164.objecter ms_handle_connect 0x7f2abc06c820 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f2ad4059100 con 0x7f2adc146080 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 10 client.16164.objecter ms_dispatch 0x7f2adc000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 3 client.16164.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 3 client.16164.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 10 client.16164.objecter checking command 1 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7f2abc074150 0x7f2abc0765f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 20 client.16164.objecter _get_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 20 client.16164.objecter put_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 15 client.16164.objecter _session_command_op_remove -1 1 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 20 client.16164.objecter _get_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 20 client.16164.objecter get_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 15 client.16164.objecter _session_command_op_assign 3 1 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 20 client.16164.objecter put_session s=0x7f2abc073a80 osd=3 4 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 10 client.16164.objecter _send_command 1 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 --> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2abc076cc0 con 0x7f2abc074150 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 20 client.16164.objecter dump_active .. 0 homeless 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ad4054d40 con 0x7f2adc146080 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.710+0000 7f2ad2ffd640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.711+0000 7f2ad3fff640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7f2abc074150 0x7f2abc0765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.711+0000 7f2ad3fff640 10 monclient: get_auth_request con 0x7f2abc074150 auth_method 0 2023-12-07T07:55:52.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.714+0000 7f2ad2ffd640 10 client.16164.objecter ms_handle_connect 0x7f2abc074150 2023-12-07T07:55:52.716 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.714+0000 7f2ad3fff640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7f2abc074150 0x7f2abc0765f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.757+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== osd.3 v2:172.21.15.97:6810/929626392 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2abc076cc0 con 0x7f2abc074150 2023-12-07T07:55:52.759 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.757+0000 7f2ad2ffd640 10 client.16164.objecter ms_dispatch 0x7f2adc000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:52.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.757+0000 7f2ad2ffd640 10 client.16164.objecter _finish_command 1 = system:0 2023-12-07T07:55:52.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.757+0000 7f2ad2ffd640 20 client.16164.objecter put_session s=0x7f2abc073a80 osd=3 4 2023-12-07T07:55:52.760 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.757+0000 7f2ad2ffd640 15 client.16164.objecter _session_command_op_remove 3 1 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 10 client.16164.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 15 client.16164.objecter _session_command_op_assign -1 2 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 20 client.16164.objecter _get_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 20 client.16164.objecter put_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 20 client.16164.objecter _get_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 15 client.16164.objecter _session_command_op_remove -1 2 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 20 client.16164.objecter get_session s=0x7f2abc073a80 osd=3 3 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 15 client.16164.objecter _session_command_op_assign 3 2 2023-12-07T07:55:52.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 20 client.16164.objecter put_session s=0x7f2abc073a80 osd=3 4 2023-12-07T07:55:52.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 10 client.16164.objecter _send_command 2 2023-12-07T07:55:52.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.770+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 --> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2adc0b2800 con 0x7f2abc074150 2023-12-07T07:55:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ad2ffd640 1 -- 172.21.15.97:0/1242735785 <== osd.3 v2:172.21.15.97:6810/929626392 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2ac4004040 con 0x7f2abc074150 2023-12-07T07:55:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ad2ffd640 10 client.16164.objecter ms_dispatch 0x7f2adc000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:52.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ad2ffd640 10 client.16164.objecter _finish_command 2 = system:0 2023-12-07T07:55:52.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ad2ffd640 20 client.16164.objecter put_session s=0x7f2abc073a80 osd=3 4 2023-12-07T07:55:52.787 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ad2ffd640 15 client.16164.objecter _session_command_op_remove 3 2 2023-12-07T07:55:52.787 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:52.787 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:52.788 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 10 librados: watch_flush enter 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 10 librados: watch_flush exit 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 10 client.16164.objecter close_session for osd.3 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7f2abc074150 msgr2=0x7f2abc0765f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6810/929626392,v1:172.21.15.97:6811/929626392] conn(0x7f2abc074150 0x7f2abc0765f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 20 client.16164.objecter put_session s=0x7f2abc073a80 osd=3 1 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 20 client.16164.objecter shutdown clearing up homeless session... 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 10 client.16164.objecter successfully canceled tick 2023-12-07T07:55:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2abc06c820 msgr2=0x7f2abc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f2abc06c820 0x7f2abc06ece0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2acc000f60 tx=0x7f2acc0079b0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 10 monclient: shutdown 2023-12-07T07:55:52.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:52.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 msgr2=0x7f2adc141910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 --2- 172.21.15.97:0/1242735785 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2adc146080 0x7f2adc141910 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400b970 tx=0x7f2ad400be30 comp rx=0 tx=0).stop 2023-12-07T07:55:52.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2adad9d640 1 -- 172.21.15.97:0/1242735785 reap_dead start 2023-12-07T07:55:52.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 shutdown_connections 2023-12-07T07:55:52.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 >> 172.21.15.97:0/1242735785 conn(0x7f2adc0b41f0 msgr2=0x7f2adc1433d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:52.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 shutdown_connections 2023-12-07T07:55:52.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.784+0000 7f2ae0bf0640 1 -- 172.21.15.97:0/1242735785 wait complete. 2023-12-07T07:55:52.791 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.785+0000 7f2ae0bf0640 1 librados: shutdown 2023-12-07T07:55:52.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-07T07:55:52.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.898+0000 7f6961209640 10 monclient: get_monmap_and_config 2023-12-07T07:55:52.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.898+0000 7f6961209640 10 monclient: build_initial_monmap 2023-12-07T07:55:52.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.898+0000 7f6961209640 10 monclient: monmap: 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:52.899184+0000 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:52.899184+0000 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:52.901 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.898+0000 7f6961209640 1 Processor -- start 2023-12-07T07:55:52.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.898+0000 7f6961209640 1 -- start start 2023-12-07T07:55:52.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.898+0000 7f6961209640 10 monclient: init 2023-12-07T07:55:52.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:52.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-12-07T07:55:52.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f695c0a8a30 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:52.903 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 0x7f695c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient(hunting): picked mon.noname-b con 0x7f695c142bf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f695c0a8e10 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c0aac10 con 0x7f695c0a8a30 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c0aaef0 con 0x7f695c0a8e10 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0aaa10 con 0x7f695c142bf0 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f6961209640 10 monclient(hunting): authenticate will time out at 2675.277658s 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f695ad9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f695c0a8a30 0x7f695c1424e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47080/0 (socket says 172.21.15.97:47080) 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.899+0000 7f695ad9d640 1 -- 172.21.15.97:0/3501597486 learned_addr learned my addr 172.21.15.97:0/3501597486 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:52.904 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.901+0000 7f695259c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 0x7f695c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984451750 0 0) 0x7f695c0aac10 con 0x7f695c0a8a30 2023-12-07T07:55:52.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): _negotiate 2023-12-07T07:55:52.905 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): my global_id is 16167 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6948003520 con 0x7f695c0a8a30 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695259c640 10 monclient(hunting): get_auth_request con 0x7f695c142bf0 auth_method 0 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345615692 0 0) 0x7f695c0aaef0 con 0x7f695c0a8e10 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): _negotiate 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 10 monclient(hunting): my global_id is 20212 2023-12-07T07:55:52.906 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.902+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f695c0aac10 con 0x7f695c0a8e10 2023-12-07T07:55:52.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.903+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:52.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.903+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:52.907 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.903+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:52.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239486743 0 0) 0x7f695c0aac10 con 0x7f695c0a8e10 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 msgr2=0x7f695c13ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 0x7f695c13ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 >> v1:172.21.15.97:6789/0 conn(0x7f695c0a8a30 legacy=0x7f695c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: _finish_hunting 0 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 monclient: found mon.noname-c-legacy 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0a97f0 con 0x7f695c0a8e10 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: _finish_auth 0 2023-12-07T07:55:52.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: _check_auth_tickets 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f6961209640 5 monclient: authenticate success, global_id 20212 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f6961209640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2176119509 0 0) 0x7f69540019f0 con 0x7f695c0a8e10 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: dump: 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 >> v1:172.21.15.154:6789/0 conn(0x7f695c0a8e10 legacy=0x7f695c13f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.905+0000 7f695a59c640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6948003b00 0x7f6948003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): picked mon.b con 0x7f6948003b00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69480047c0 0x7f6948004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): picked mon.a con 0x7f69480047c0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 0x7f6948005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): picked mon.c con 0x7f695c142bf0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0aaa10 con 0x7f69480047c0 2023-12-07T07:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694800ba30 con 0x7f6948003b00 2023-12-07T07:55:52.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694800bbc0 con 0x7f695c142bf0 2023-12-07T07:55:52.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:52.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f695a59c640 10 monclient(hunting): _finish_auth 0 2023-12-07T07:55:52.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.906+0000 7f6961209640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.907+0000 7f695259c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69480047c0 0x7f6948004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.907+0000 7f695259c640 10 monclient(hunting): get_auth_request con 0x7f69480047c0 auth_method 0 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.907+0000 7f695259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.907+0000 7f695259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.907+0000 7f695259c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.908+0000 7f695b59e640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 0x7f6948005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.908+0000 7f695ad9d640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6948003b00 0x7f6948003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.908+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.908+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.908+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.909+0000 7f695ad9d640 10 monclient(hunting): get_auth_request con 0x7f6948003b00 auth_method 0 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.909+0000 7f695ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:52.914 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.909+0000 7f695ad9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.909+0000 7f695ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.909+0000 7f695ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.909+0000 7f695ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.909+0000 7f695ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 10 monclient(hunting): handle_auth_done global_id 20212 payload 293 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 1 -- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 msgr2=0x7f6948005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f695c142bf0 0x7f6948005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 1 -- 172.21.15.97:0/3501597486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6948003b00 msgr2=0x7f6948003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6948003b00 0x7f6948003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 10 monclient: _finish_hunting 0 2023-12-07T07:55:52.915 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 1 monclient: found mon.a 2023-12-07T07:55:52.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:52.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:52.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 1 -- 172.21.15.97:0/3501597486 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0aac10 con 0x7f69480047c0 2023-12-07T07:55:52.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 10 monclient: _finish_auth 0 2023-12-07T07:55:52.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 10 monclient: _check_auth_tickets 2023-12-07T07:55:52.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.910+0000 7f695259c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:52.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.912+0000 7f695259c640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69480047c0 0x7f6948004c40 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f6944009ac0 tx=0x7f69480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa605be3af0a448 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.914+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694402c660 con 0x7f69480047c0 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.914+0000 7f695a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.914+0000 7f695a59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.914+0000 7f695a59c640 10 monclient: dump: 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.917 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.914+0000 7f6961209640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694402c7c0 con 0x7f69480047c0 2023-12-07T07:55:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f695a59c640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f695a59c640 1 -- 172.21.15.97:0/3501597486 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694402cb50 con 0x7f69480047c0 2023-12-07T07:55:52.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f695a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f695a59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f695a59c640 10 monclient: dump: 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:52.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 4 set_mon_vals no callback set 2023-12-07T07:55:52.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 10 monclient: shutdown 2023-12-07T07:55:52.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:52.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 1 -- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69480047c0 msgr2=0x7f6948004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 1 --2- 172.21.15.97:0/3501597486 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69480047c0 0x7f6948004c40 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f6944009ac0 tx=0x7f69480057f0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f695b59e640 1 -- 172.21.15.97:0/3501597486 reap_dead start 2023-12-07T07:55:52.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 1 -- 172.21.15.97:0/3501597486 shutdown_connections 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.916+0000 7f6961209640 1 -- 172.21.15.97:0/3501597486 >> 172.21.15.97:0/3501597486 conn(0x7f695c061a30 msgr2=0x7f695c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.917+0000 7f6961209640 1 -- 172.21.15.97:0/3501597486 shutdown_connections 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.917+0000 7f6961209640 1 -- 172.21.15.97:0/3501597486 wait complete. 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.917+0000 7f6961209640 10 monclient: build_initial_monmap 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.917+0000 7f6961209640 10 monclient: monmap: 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:52.923 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 librados: starting msgr at 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 librados: starting objecter 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 Processor -- start 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 -- start start 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 librados: setting wanted keys 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 librados: calling monclient init 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient: init 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69480047c0 0x7f695c141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): picked mon.noname-c con 0x7f69480047c0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695c144900 0x7f695c144d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): picked mon.noname-a con 0x7f695c144900 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:52.924 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f695c1452a0 0x7f695c1da390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): picked mon.noname-b con 0x7f695c1452a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c140c10 con 0x7f695c144900 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c143d90 con 0x7f695c1452a0 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c144000 con 0x7f69480047c0 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.918+0000 7f6961209640 10 monclient(hunting): authenticate will time out at 2675.297256s 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695259c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695c144900 0x7f695c144d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f695c1452a0 0x7f695c1da390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f695c1452a0 0x7f695c1da390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:42088/0 (socket says 172.21.15.97:42088) 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695b59e640 1 -- 172.21.15.97:0/2732175842 learned_addr learned my addr 172.21.15.97:0/2732175842 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695b59e640 10 monclient(hunting): get_auth_request con 0x7f695c1452a0 auth_method 0 2023-12-07T07:55:52.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695b59e640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695b59e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695259c640 10 monclient(hunting): get_auth_request con 0x7f695c144900 auth_method 0 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695259c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.919+0000 7f695259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.920+0000 7f695b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.920+0000 7f695b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.920+0000 7f695b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.920+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.920+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:52.926 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.920+0000 7f695259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.921+0000 7f695ad9d640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69480047c0 0x7f695c141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 10 monclient(hunting): handle_auth_done global_id 20215 payload 824 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 1 -- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69480047c0 msgr2=0x7f695c141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69480047c0 0x7f695c141710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 1 -- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695c144900 msgr2=0x7f695c144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695c144900 0x7f695c144d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 10 monclient: _finish_hunting 0 2023-12-07T07:55:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 1 monclient: found mon.noname-b 2023-12-07T07:55:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.922+0000 7f695b59e640 1 -- 172.21.15.97:0/2732175842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c13fd20 con 0x7f695c1452a0 2023-12-07T07:55:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.924+0000 7f695b59e640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f695c1452a0 0x7f695c1da390 secure :-1 s=READY pgs=4867 cs=0 l=1 rev1=1 crypto rx=0x7f69540012e0 tx=0x7f695400df70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.924+0000 7f6953fff640 10 client.?.objecter ms_handle_connect 0x7f695c1452a0 2023-12-07T07:55:52.928 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.924+0000 7f6953fff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 1 -- 172.21.15.97:0/2732175842 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954003d80 con 0x7f695c1452a0 2023-12-07T07:55:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.930 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: dump: 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: _finish_auth 0 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: _check_auth_tickets 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:52.931 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 1 -- 172.21.15.97:0/2732175842 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695400b740 con 0x7f695c1452a0 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 1 -- 172.21.15.97:0/2732175842 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695400baf0 con 0x7f695c1452a0 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6953fff640 10 monclient: dump: 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:52.932 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f695959a640 4 set_mon_vals no callback set 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6961209640 5 monclient: authenticate success, global_id 20215 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6961209640 10 monclient: _renew_subs 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6961209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.927+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c141c50 con 0x7f695c1452a0 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 10 client.20215.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 10 monclient: _renew_subs 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c1daab0 con 0x7f695c1452a0 2023-12-07T07:55:52.933 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 1 librados: init done 2023-12-07T07:55:52.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:52.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:52.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:52.934 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.928+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c067ac0 con 0x7f695c1452a0 2023-12-07T07:55:52.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.947+0000 7f6953fff640 1 -- 172.21.15.97:0/2732175842 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f695400bc50 con 0x7f695c1452a0 2023-12-07T07:55:52.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.947+0000 7f6953fff640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693c06c850 0x7f693c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:52.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.947+0000 7f695ad9d640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693c06c850 0x7f693c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:52.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.947+0000 7f695ad9d640 10 monclient: get_auth_request con 0x7f693c06c850 auth_method 0 2023-12-07T07:55:52.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.948+0000 7f6953fff640 10 client.20215.objecter ms_handle_connect 0x7f693c06c850 2023-12-07T07:55:52.950 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.948+0000 7f695ad9d640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693c06c850 0x7f693c06ed10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f694c007fd0 tx=0x7f694c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:52.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.948+0000 7f6953fff640 1 -- 172.21.15.97:0/2732175842 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f6954058420 con 0x7f695c1452a0 2023-12-07T07:55:52.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.948+0000 7f6953fff640 10 client.20215.objecter ms_dispatch 0x7f695c000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:52.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.948+0000 7f6953fff640 3 client.20215.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:52.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.948+0000 7f6953fff640 3 client.20215.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:52.951 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:52.948+0000 7f6953fff640 20 client.20215.objecter dump_active .. 0 homeless 2023-12-07T07:55:53.185 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.182+0000 7f6953fff640 1 -- 172.21.15.97:0/2732175842 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6954054760 con 0x7f695c1452a0 2023-12-07T07:55:53.185 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.182+0000 7f6953fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:53.185 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.182+0000 7f6953fff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:53.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.360+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f695c13f940 con 0x7f693c06c850 2023-12-07T07:55:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.360+0000 7f6953fff640 1 -- 172.21.15.97:0/2732175842 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f694c0014b0 con 0x7f693c06c850 2023-12-07T07:55:53.363 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 3 to deep-scrub 2023-12-07T07:55:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 10 librados: watch_flush enter 2023-12-07T07:55:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 10 librados: watch_flush exit 2023-12-07T07:55:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 20 client.20215.objecter shutdown clearing up homeless session... 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 10 client.20215.objecter successfully canceled tick 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693c06c850 msgr2=0x7f693c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693c06c850 0x7f693c06ed10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f694c007fd0 tx=0x7f694c009040 comp rx=0 tx=0).stop 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 10 monclient: shutdown 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f695c1452a0 msgr2=0x7f695c1da390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.362+0000 7f6961209640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f695c1452a0 0x7f695c1da390 secure :-1 s=READY pgs=4867 cs=0 l=1 rev1=1 crypto rx=0x7f69540012e0 tx=0x7f695400df70 comp rx=0 tx=0).stop 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 shutdown_connections 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f693c06c850 0x7f693c06ed10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f695c1452a0 0x7f695c1da390 unknown :-1 s=CLOSED pgs=4867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f695c144900 0x7f695c144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 --2- 172.21.15.97:0/2732175842 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69480047c0 0x7f695c141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 >> 172.21.15.97:0/2732175842 conn(0x7f695c061a30 msgr2=0x7f695c13ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 shutdown_connections 2023-12-07T07:55:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 -- 172.21.15.97:0/2732175842 wait complete. 2023-12-07T07:55:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.363+0000 7f6961209640 1 librados: shutdown 2023-12-07T07:55:53.376 INFO:tasks.ceph:Scrubbing osd.4 2023-12-07T07:55:53.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.486+0000 7fee0aa90640 10 monclient: get_monmap_and_config 2023-12-07T07:55:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.486+0000 7fee0aa90640 10 monclient: build_initial_monmap 2023-12-07T07:55:53.488 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.486+0000 7fee0aa90640 10 monclient: monmap: 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:53.487245+0000 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:53.487245+0000 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:53.489 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.486+0000 7fee0aa90640 1 Processor -- start 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.486+0000 7fee0aa90640 1 -- start start 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.486+0000 7fee0aa90640 10 monclient: init 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee04145840 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fee04146080 0x7fee0413f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient(hunting): picked mon.noname-a con 0x7fee04146080 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fee0413fa50 0x7fee0414c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient(hunting): picked mon.noname-c con 0x7fee0413fa50 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee040aa850 con 0x7fee04145840 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee04145c20 con 0x7fee04146080 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee040b2800 con 0x7fee0413fa50 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.487+0000 7fee0aa90640 10 monclient(hunting): authenticate will time out at 2675.866195s 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.488+0000 7fee0902d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fee0413fa50 0x7fee0414c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.488+0000 7fee0902d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fee0413fa50 0x7fee0414c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:42092/0 (socket says 172.21.15.97:42092) 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.488+0000 7fee0902d640 1 -- 172.21.15.97:0/4128122986 learned_addr learned my addr 172.21.15.97:0/4128122986 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.488+0000 7fee0902d640 10 monclient(hunting): get_auth_request con 0x7fee0413fa50 auth_method 0 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.488+0000 7fee0902d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.488+0000 7fee0902d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.488+0000 7fee0902d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.489+0000 7fee0902d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.489+0000 7fee0902d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.489+0000 7fee0902d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee03fff640 1 --2- 172.21.15.97:0/4128122986 >> v2:172.21.15.97:3300/0 conn(0x7fee04146080 0x7fee0413f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee03fff640 10 monclient(hunting): get_auth_request con 0x7fee04146080 auth_method 0 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee03fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee03fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 10 monclient(hunting): handle_auth_done global_id 20218 payload 274 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 1 -- 172.21.15.97:0/4128122986 >> v2:172.21.15.97:3300/0 conn(0x7fee04146080 msgr2=0x7fee0413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 1 --2- 172.21.15.97:0/4128122986 >> v2:172.21.15.97:3300/0 conn(0x7fee04146080 0x7fee0413f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 1 -- 172.21.15.97:0/4128122986 >> v1:172.21.15.97:6789/0 conn(0x7fee04145840 legacy=0x7fee040b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 10 monclient: _finish_hunting 0 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 1 monclient: found mon.noname-c 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:53.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.490+0000 7fee0902d640 1 -- 172.21.15.97:0/4128122986 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee040a9260 con 0x7fee0413fa50 2023-12-07T07:55:53.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.492+0000 7fee0902d640 1 --2- 172.21.15.97:0/4128122986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 0x7fee0414c960 secure :-1 s=READY pgs=4868 cs=0 l=1 rev1=1 crypto rx=0x7fee040aa490 tx=0x7fedfc02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9808d4aef7656f14 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 1 -- 172.21.15.97:0/4128122986 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc02c9f0 con 0x7fee0413fa50 2023-12-07T07:55:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.498 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: dump: 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: _finish_auth 0 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: _check_auth_tickets 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:53.499 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 1 -- 172.21.15.97:0/4128122986 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc02cb50 con 0x7fee0413fa50 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 5 monclient: authenticate success, global_id 20218 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 4 set_mon_vals no callback set 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 1 -- 172.21.15.97:0/4128122986 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0343f0 con 0x7fee0413fa50 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee037fe640 10 monclient: dump: 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.500 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.501 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.501 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 10 monclient: shutdown 2023-12-07T07:55:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:53.503 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 1 -- 172.21.15.97:0/4128122986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 msgr2=0x7fee0414c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.496+0000 7fee0aa90640 1 --2- 172.21.15.97:0/4128122986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 0x7fee0414c960 secure :-1 s=READY pgs=4868 cs=0 l=1 rev1=1 crypto rx=0x7fee040aa490 tx=0x7fedfc02be50 comp rx=0 tx=0).stop 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 -- 172.21.15.97:0/4128122986 shutdown_connections 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 --2- 172.21.15.97:0/4128122986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 0x7fee0414c960 unknown :-1 s=CLOSED pgs=4868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 --2- 172.21.15.97:0/4128122986 >> v2:172.21.15.97:3300/0 conn(0x7fee04146080 0x7fee0413f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 -- 172.21.15.97:0/4128122986 >> 172.21.15.97:0/4128122986 conn(0x7fee040b41f0 msgr2=0x7fee040b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 -- 172.21.15.97:0/4128122986 shutdown_connections 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 -- 172.21.15.97:0/4128122986 wait complete. 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 10 monclient: build_initial_monmap 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 10 monclient: monmap: 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:53.504 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 librados: starting msgr at 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.497+0000 7fee0aa90640 1 librados: starting objecter 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 Processor -- start 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 -- start start 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 librados: setting wanted keys 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 librados: calling monclient init 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient: init 2023-12-07T07:55:53.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 0x7fee041dac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): picked mon.noname-b con 0x7fee0413fa50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee04145840 0x7fee041db3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): picked mon.noname-c con 0x7fee04145840 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04146080 0x7fee041cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): picked mon.noname-a con 0x7fee04146080 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041db910 con 0x7fee04146080 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee041cf160 con 0x7fee0413fa50 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee040ad5d0 con 0x7fee04145840 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:53.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.498+0000 7fee0aa90640 10 monclient(hunting): authenticate will time out at 2675.877046s 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04146080 0x7fee041cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 0x7fee041dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee03fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee04145840 0x7fee041db3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04146080 0x7fee041cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45904/0 (socket says 172.21.15.97:45904) 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 1 -- 172.21.15.97:0/1209404407 learned_addr learned my addr 172.21.15.97:0/1209404407 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 10 monclient(hunting): get_auth_request con 0x7fee04146080 auth_method 0 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 10 monclient(hunting): get_auth_request con 0x7fee0413fa50 auth_method 0 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee03fff640 10 monclient(hunting): get_auth_request con 0x7fee04145840 auth_method 0 2023-12-07T07:55:53.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee03fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee03fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0882c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.499+0000 7fee0902d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.500+0000 7fee03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.500+0000 7fee03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.500+0000 7fee03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 10 monclient(hunting): handle_auth_done global_id 20221 payload 824 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 1 -- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee04145840 msgr2=0x7fee041db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee04145840 0x7fee041db3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 1 -- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04146080 msgr2=0x7fee041cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee04146080 0x7fee041cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 10 monclient: _finish_hunting 0 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 1 monclient: found mon.noname-b 2023-12-07T07:55:53.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:53.510 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:53.510 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.502+0000 7fee0882c640 1 -- 172.21.15.97:0/1209404407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc009b30 con 0x7fee0413fa50 2023-12-07T07:55:53.510 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.504+0000 7fee0882c640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 0x7fee041dac30 secure :-1 s=READY pgs=4869 cs=0 l=1 rev1=1 crypto rx=0x7fedf800c410 tx=0x7fedf800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:53.510 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.504+0000 7fee017fa640 10 client.?.objecter ms_handle_connect 0x7fee0413fa50 2023-12-07T07:55:53.510 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.504+0000 7fee017fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:53.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8005230 con 0x7fee0413fa50 2023-12-07T07:55:53.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:53.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: dump: 2023-12-07T07:55:53.511 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: _finish_auth 0 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: _check_auth_tickets 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8005390 con 0x7fee0413fa50 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8004820 con 0x7fee0413fa50 2023-12-07T07:55:53.512 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 5 monclient: authenticate success, global_id 20221 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 monclient: _renew_subs 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee041ce1d0 con 0x7fee0413fa50 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee02ffd640 4 set_mon_vals no callback set 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee017fa640 10 monclient: dump: 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.513 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 client.20221.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 monclient: _renew_subs 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee041cc960 con 0x7fee0413fa50 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 1 librados: init done 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 client.20221.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 15 client.20221.objecter _session_command_op_assign -1 1 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 20 client.20221.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 client.20221.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:53.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.508+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee041464b0 con 0x7fee0413fa50 2023-12-07T07:55:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.527+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fedf8004980 con 0x7fee0413fa50 2023-12-07T07:55:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.527+0000 7fee017fa640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fede806c820 0x7fede806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.530 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.527+0000 7fee03fff640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fede806c820 0x7fede806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.527+0000 7fee03fff640 10 monclient: get_auth_request con 0x7fede806c820 auth_method 0 2023-12-07T07:55:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.527+0000 7fee03fff640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fede806c820 0x7fede806ece0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee041432e0 tx=0x7fedf0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.527+0000 7fee017fa640 10 client.20221.objecter ms_handle_connect 0x7fede806c820 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fedf8057bc0 con 0x7fee0413fa50 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 10 client.20221.objecter ms_dispatch 0x7fee04000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 3 client.20221.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 3 client.20221.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 10 client.20221.objecter checking command 1 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fede8074150 0x7fede80765f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 20 client.20221.objecter _get_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 20 client.20221.objecter put_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 15 client.20221.objecter _session_command_op_remove -1 1 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 20 client.20221.objecter _get_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 20 client.20221.objecter get_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 15 client.20221.objecter _session_command_op_assign 4 1 2023-12-07T07:55:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 20 client.20221.objecter put_session s=0x7fede8073a80 osd=4 4 2023-12-07T07:55:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 10 client.20221.objecter _send_command 1 2023-12-07T07:55:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 --> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fede8076cc0 con 0x7fede8074150 2023-12-07T07:55:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee017fa640 20 client.20221.objecter dump_active .. 0 homeless 2023-12-07T07:55:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.529+0000 7fee0902d640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fede8074150 0x7fede80765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.530+0000 7fee0902d640 10 monclient: get_auth_request con 0x7fede8074150 auth_method 0 2023-12-07T07:55:53.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.532+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf8053f00 con 0x7fee0413fa50 2023-12-07T07:55:53.534 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.532+0000 7fee017fa640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.532+0000 7fee017fa640 10 client.20221.objecter ms_handle_connect 0x7fede8074150 2023-12-07T07:55:53.536 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.532+0000 7fee0902d640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fede8074150 0x7fede80765f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:53.559 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.556+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== osd.4 v2:172.21.15.154:6808/1781054720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fede8076cc0 con 0x7fede8074150 2023-12-07T07:55:53.559 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.556+0000 7fee017fa640 10 client.20221.objecter ms_dispatch 0x7fee04000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:53.559 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.556+0000 7fee017fa640 10 client.20221.objecter _finish_command 1 = system:0 2023-12-07T07:55:53.559 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.556+0000 7fee017fa640 20 client.20221.objecter put_session s=0x7fede8073a80 osd=4 4 2023-12-07T07:55:53.559 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.556+0000 7fee017fa640 15 client.20221.objecter _session_command_op_remove 4 1 2023-12-07T07:55:53.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 10 client.20221.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:53.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 15 client.20221.objecter _session_command_op_assign -1 2 2023-12-07T07:55:53.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 20 client.20221.objecter _get_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 20 client.20221.objecter put_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 20 client.20221.objecter _get_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.572 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 15 client.20221.objecter _session_command_op_remove -1 2 2023-12-07T07:55:53.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 20 client.20221.objecter get_session s=0x7fede8073a80 osd=4 3 2023-12-07T07:55:53.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 15 client.20221.objecter _session_command_op_assign 4 2 2023-12-07T07:55:53.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 20 client.20221.objecter put_session s=0x7fede8073a80 osd=4 4 2023-12-07T07:55:53.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 10 client.20221.objecter _send_command 2 2023-12-07T07:55:53.573 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.570+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 --> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fee040aad80 con 0x7fede8074150 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.583+0000 7fee017fa640 1 -- 172.21.15.97:0/1209404407 <== osd.4 v2:172.21.15.154:6808/1781054720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fedfc02c9b0 con 0x7fede8074150 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.583+0000 7fee017fa640 10 client.20221.objecter ms_dispatch 0x7fee04000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.583+0000 7fee017fa640 10 client.20221.objecter _finish_command 2 = system:0 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.583+0000 7fee017fa640 20 client.20221.objecter put_session s=0x7fede8073a80 osd=4 4 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.583+0000 7fee017fa640 15 client.20221.objecter _session_command_op_remove 4 2 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:53.586 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 10 librados: watch_flush enter 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 10 librados: watch_flush exit 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 10 client.20221.objecter close_session for osd.4 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fede8074150 msgr2=0x7fede80765f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:6808/1781054720,v1:172.21.15.154:6809/1781054720] conn(0x7fede8074150 0x7fede80765f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 20 client.20221.objecter put_session s=0x7fede8073a80 osd=4 1 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 20 client.20221.objecter shutdown clearing up homeless session... 2023-12-07T07:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 10 client.20221.objecter successfully canceled tick 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fede806c820 msgr2=0x7fede806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fede806c820 0x7fede806ece0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee041432e0 tx=0x7fedf0008040 comp rx=0 tx=0).stop 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 10 monclient: shutdown 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 msgr2=0x7fee041dac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 --2- 172.21.15.97:0/1209404407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0413fa50 0x7fee041dac30 secure :-1 s=READY pgs=4869 cs=0 l=1 rev1=1 crypto rx=0x7fedf800c410 tx=0x7fedf800c8d0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0902d640 1 -- 172.21.15.97:0/1209404407 reap_dead start 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 shutdown_connections 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 >> 172.21.15.97:0/1209404407 conn(0x7fee040b41f0 msgr2=0x7fee040b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 shutdown_connections 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 -- 172.21.15.97:0/1209404407 wait complete. 2023-12-07T07:55:53.589 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.584+0000 7fee0aa90640 1 librados: shutdown 2023-12-07T07:55:53.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-07T07:55:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.696+0000 7f34daea2640 10 monclient: get_monmap_and_config 2023-12-07T07:55:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.696+0000 7f34daea2640 10 monclient: build_initial_monmap 2023-12-07T07:55:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.696+0000 7f34daea2640 10 monclient: monmap: 2023-12-07T07:55:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.699 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:53.698000+0000 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:53.698000+0000 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:53.700 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.697+0000 7f34daea2640 1 Processor -- start 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.697+0000 7f34daea2640 1 -- start start 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.697+0000 7f34daea2640 10 monclient: init 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34d4060f20 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f34d4061300 0x7f34d41430f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient(hunting): picked mon.noname-a con 0x7f34d4061300 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34d40ae830 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40a8660 con 0x7f34d40ae830 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40a8940 con 0x7f34d4060f20 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d40af300 con 0x7f34d4061300 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34daea2640 10 monclient(hunting): authenticate will time out at 2676.076712s 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34d8c3e640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f34d4060f20 0x7f34d40ad660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:53864/0 (socket says 172.21.15.97:53864) 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.698+0000 7f34d8c3e640 1 -- 172.21.15.97:0/3756668963 learned_addr learned my addr 172.21.15.97:0/3756668963 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.699+0000 7f34d3fff640 1 --2- 172.21.15.97:0/3756668963 >> v2:172.21.15.97:3300/0 conn(0x7f34d4061300 0x7f34d41430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.699+0000 7f34d3fff640 10 monclient(hunting): get_auth_request con 0x7f34d4061300 auth_method 0 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.699+0000 7f34d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.699+0000 7f34d3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.703 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.699+0000 7f34d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d37fe640 1 -- 172.21.15.97:0/3756668963 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858599354 0 0) 0x7f34d40a8940 con 0x7f34d4060f20 2023-12-07T07:55:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d37fe640 10 monclient(hunting): _negotiate 2023-12-07T07:55:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d37fe640 10 monclient(hunting): my global_id is 20224 2023-12-07T07:55:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d37fe640 1 -- 172.21.15.97:0/3756668963 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc003520 con 0x7f34d4060f20 2023-12-07T07:55:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.701+0000 7f34d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 10 monclient(hunting): handle_auth_done global_id 16182 payload 274 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 1 -- 172.21.15.97:0/3756668963 >> v1:172.21.15.154:6789/0 conn(0x7f34d4060f20 legacy=0x7f34d40ad660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 1 -- 172.21.15.97:0/3756668963 >> v1:172.21.15.97:6789/0 conn(0x7f34d40ae830 legacy=0x7f34d4143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 1 monclient: found mon.noname-a 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:53.706 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.703+0000 7f34d3fff640 1 -- 172.21.15.97:0/3756668963 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d40a8c20 con 0x7f34d4061300 2023-12-07T07:55:53.708 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.706+0000 7f34d3fff640 1 --2- 172.21.15.97:0/3756668963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4061300 0x7f34d41430f0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7f34c80097a0 tx=0x7f34c802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cfc5196beee29 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 1 -- 172.21.15.97:0/3756668963 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8030610 con 0x7f34d4061300 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: dump: 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.713 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: _finish_auth 0 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: _check_auth_tickets 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 1 -- 172.21.15.97:0/3756668963 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c8030770 con 0x7f34d4061300 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 1 -- 172.21.15.97:0/3756668963 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8030b00 con 0x7f34d4061300 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34daea2640 5 monclient: authenticate success, global_id 16182 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34daea2640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34daea2640 4 set_mon_vals no callback set 2023-12-07T07:55:53.714 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.710+0000 7f34d37fe640 10 monclient: dump: 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.715 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 10 monclient: shutdown 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 1 -- 172.21.15.97:0/3756668963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4061300 msgr2=0x7f34d41430f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 1 --2- 172.21.15.97:0/3756668963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4061300 0x7f34d41430f0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7f34c80097a0 tx=0x7f34c802b960 comp rx=0 tx=0).stop 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 1 -- 172.21.15.97:0/3756668963 shutdown_connections 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 1 --2- 172.21.15.97:0/3756668963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4061300 0x7f34d41430f0 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 1 -- 172.21.15.97:0/3756668963 >> 172.21.15.97:0/3756668963 conn(0x7f34d40b3e30 msgr2=0x7f34d40b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 1 -- 172.21.15.97:0/3756668963 shutdown_connections 2023-12-07T07:55:53.719 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.711+0000 7f34daea2640 1 -- 172.21.15.97:0/3756668963 wait complete. 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 10 monclient: build_initial_monmap 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 10 monclient: monmap: 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 1 librados: starting msgr at 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 1 librados: starting objecter 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 1 Processor -- start 2023-12-07T07:55:53.720 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 1 -- start start 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 1 librados: setting wanted keys 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 1 librados: calling monclient init 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.712+0000 7f34daea2640 10 monclient: init 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4060f20 0x7f34d413faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): picked mon.noname-a con 0x7f34d4060f20 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34d40ae830 0x7f34d4134050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): picked mon.noname-c con 0x7f34d40ae830 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34d4070330 0x7f34d4134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): picked mon.noname-b con 0x7f34d4070330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4134ad0 con 0x7f34d4060f20 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.721 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4140600 con 0x7f34d4070330 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4148810 con 0x7f34d40ae830 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.713+0000 7f34daea2640 10 monclient(hunting): authenticate will time out at 2676.091925s 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4060f20 0x7f34d413faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d3fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34d40ae830 0x7f34d4134050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4060f20 0x7f34d413faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45920/0 (socket says 172.21.15.97:45920) 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 1 -- 172.21.15.97:0/2363160789 learned_addr learned my addr 172.21.15.97:0/2363160789 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 10 monclient(hunting): get_auth_request con 0x7f34d4060f20 auth_method 0 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d943f640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34d4070330 0x7f34d4134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d3fff640 10 monclient(hunting): get_auth_request con 0x7f34d40ae830 auth_method 0 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d943f640 10 monclient(hunting): get_auth_request con 0x7f34d4070330 auth_method 0 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d943f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d943f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d943f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.714+0000 7f34d8c3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.715+0000 7f34d943f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.715+0000 7f34d943f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.715+0000 7f34d943f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.715+0000 7f34d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.715+0000 7f34d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.715+0000 7f34d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 10 monclient(hunting): handle_auth_done global_id 16185 payload 824 2023-12-07T07:55:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 1 -- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34d40ae830 msgr2=0x7f34d4134050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34d40ae830 0x7f34d4134050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 1 -- 172.21.15.97:0/2363160789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34d4070330 msgr2=0x7f34d4134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34d4070330 0x7f34d4134590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 10 monclient: _finish_hunting 0 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 1 monclient: found mon.noname-a 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.717+0000 7f34d8c3e640 1 -- 172.21.15.97:0/2363160789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009400 con 0x7f34d4060f20 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.720+0000 7f34d8c3e640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4060f20 0x7f34d413faa0 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c410 tx=0x7f34c400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.720+0000 7f34d17fa640 10 client.?.objecter ms_handle_connect 0x7f34d4060f20 2023-12-07T07:55:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.720+0000 7f34d17fa640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 1 -- 172.21.15.97:0/2363160789 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4005ce0 con 0x7f34d4060f20 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: dump: 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: _finish_auth 0 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: _check_auth_tickets 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 1 -- 172.21.15.97:0/2363160789 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4005e40 con 0x7f34d4060f20 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 1 -- 172.21.15.97:0/2363160789 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40072f0 con 0x7f34d4060f20 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d17fa640 10 monclient: dump: 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 5 monclient: authenticate success, global_id 16185 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 10 monclient: _renew_subs 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4148330 con 0x7f34d4060f20 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34d2ffd640 4 set_mon_vals no callback set 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 10 client.16185.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 10 monclient: _renew_subs 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d41485e0 con 0x7f34d4060f20 2023-12-07T07:55:53.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.723+0000 7f34daea2640 1 librados: init done 2023-12-07T07:55:53.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.724+0000 7f34daea2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:53.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.724+0000 7f34daea2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:53.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.724+0000 7f34daea2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:53.730 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.724+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4149e40 con 0x7f34d4060f20 2023-12-07T07:55:53.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.743+0000 7f34d17fa640 1 -- 172.21.15.97:0/2363160789 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f34c4007510 con 0x7f34d4060f20 2023-12-07T07:55:53.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.743+0000 7f34d17fa640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f34c006c820 0x7f34c006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:53.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.743+0000 7f34d3fff640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f34c006c820 0x7f34c006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:53.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.743+0000 7f34d3fff640 10 monclient: get_auth_request con 0x7f34c006c820 auth_method 0 2023-12-07T07:55:53.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.744+0000 7f34d17fa640 10 client.16185.objecter ms_handle_connect 0x7f34c006c820 2023-12-07T07:55:53.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.744+0000 7f34d3fff640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f34c006c820 0x7f34c006ece0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f34c8036000 tx=0x7f34c803b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:53.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 1 -- 172.21.15.97:0/2363160789 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f34c4057e10 con 0x7f34d4060f20 2023-12-07T07:55:53.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 10 client.16185.objecter ms_dispatch 0x7f34d4000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:53.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 3 client.16185.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:53.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 3 client.16185.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:53.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 20 client.16185.objecter dump_active .. 0 homeless 2023-12-07T07:55:53.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 1 -- 172.21.15.97:0/2363160789 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34c4054150 con 0x7f34d4060f20 2023-12-07T07:55:53.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:53.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:53.975+0000 7f34d17fa640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.156+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f34d4149e40 con 0x7f34c006c820 2023-12-07T07:55:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.156+0000 7f34d17fa640 1 -- 172.21.15.97:0/2363160789 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f34d4149e40 con 0x7f34c006c820 2023-12-07T07:55:54.159 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 4 to deep-scrub 2023-12-07T07:55:54.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 10 librados: watch_flush enter 2023-12-07T07:55:54.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 10 librados: watch_flush exit 2023-12-07T07:55:54.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 20 client.16185.objecter shutdown clearing up homeless session... 2023-12-07T07:55:54.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 10 client.16185.objecter successfully canceled tick 2023-12-07T07:55:54.161 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f34c006c820 msgr2=0x7f34c006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f34c006c820 0x7f34c006ece0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f34c8036000 tx=0x7f34c803b040 comp rx=0 tx=0).stop 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 10 monclient: shutdown 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4060f20 msgr2=0x7f34d413faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4060f20 0x7f34d413faa0 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c410 tx=0x7f34c400c8d0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 shutdown_connections 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f34c006c820 0x7f34c006ece0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34d4070330 0x7f34d4134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34d40ae830 0x7f34d4134050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 --2- 172.21.15.97:0/2363160789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34d4060f20 0x7f34d413faa0 unknown :-1 s=CLOSED pgs=3232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 >> 172.21.15.97:0/2363160789 conn(0x7f34d40b3e30 msgr2=0x7f34d4071240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 shutdown_connections 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 -- 172.21.15.97:0/2363160789 wait complete. 2023-12-07T07:55:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.158+0000 7f34daea2640 1 librados: shutdown 2023-12-07T07:55:54.172 INFO:tasks.ceph:Scrubbing osd.5 2023-12-07T07:55:54.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.271+0000 7f0d8bc4b640 10 monclient: get_monmap_and_config 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.271+0000 7f0d8bc4b640 10 monclient: build_initial_monmap 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.271+0000 7f0d8bc4b640 10 monclient: monmap: 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:54.272745+0000 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:54.272745+0000 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:54.274 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:54.275 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:54.275 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.272+0000 7f0d8bc4b640 1 Processor -- start 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.272+0000 7f0d8bc4b640 1 -- start start 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.272+0000 7f0d8bc4b640 10 monclient: init 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d840af360 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d84141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d84138ba0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0d84072740 0x7f0d84142430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d84072740 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840af740 con 0x7f0d840af360 2023-12-07T07:55:54.277 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d840720b0 con 0x7f0d84072740 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84070be0 con 0x7f0d84138ba0 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d8bc4b640 10 monclient(hunting): authenticate will time out at 2676.651580s 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.273+0000 7f0d891e6640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d84141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0d84072740 0x7f0d84142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d84141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38812/0 (socket says 172.21.15.97:38812) 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 1 -- 172.21.15.97:0/965719347 learned_addr learned my addr 172.21.15.97:0/965719347 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 10 monclient(hunting): get_auth_request con 0x7f0d84138ba0 auth_method 0 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 10 monclient(hunting): get_auth_request con 0x7f0d84072740 auth_method 0 2023-12-07T07:55:54.278 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d891e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.279 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.274+0000 7f0d8a1e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 10 monclient(hunting): handle_auth_done global_id 17690 payload 274 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 1 -- 172.21.15.97:0/965719347 >> v2:172.21.15.154:3300/0 conn(0x7f0d84072740 msgr2=0x7f0d84142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 1 --2- 172.21.15.97:0/965719347 >> v2:172.21.15.154:3300/0 conn(0x7f0d84072740 0x7f0d84142430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 1 -- 172.21.15.97:0/965719347 >> v1:172.21.15.154:6789/0 conn(0x7f0d840af360 legacy=0x7f0d84138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 10 monclient: _finish_hunting 0 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 1 monclient: found mon.noname-b 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:55:54.280 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.276+0000 7f0d891e6640 1 -- 172.21.15.97:0/965719347 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84072320 con 0x7f0d84138ba0 2023-12-07T07:55:54.281 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.278+0000 7f0d891e6640 1 --2- 172.21.15.97:0/965719347 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d84141ef0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009870 tx=0x7f0d7402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52cfc46c4193a868 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 1 -- 172.21.15.97:0/965719347 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d74031640 con 0x7f0d84138ba0 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: dump: 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.284 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: _finish_auth 0 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: _check_auth_tickets 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 1 -- 172.21.15.97:0/965719347 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d740317a0 con 0x7f0d84138ba0 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d8bc4b640 5 monclient: authenticate success, global_id 17690 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 1 -- 172.21.15.97:0/965719347 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d74031b30 con 0x7f0d84138ba0 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d8bc4b640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d8bc4b640 4 set_mon_vals no callback set 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:54.285 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.281+0000 7f0d889e5640 10 monclient: dump: 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.286 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.289 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 10 monclient: shutdown 2023-12-07T07:55:54.289 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:54.289 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/965719347 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 msgr2=0x7f0d84141ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 --2- 172.21.15.97:0/965719347 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d84141ef0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009870 tx=0x7f0d7402b8f0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/965719347 shutdown_connections 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 --2- 172.21.15.97:0/965719347 >> v2:172.21.15.154:3300/0 conn(0x7f0d84072740 0x7f0d84142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 --2- 172.21.15.97:0/965719347 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d84141ef0 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/965719347 >> 172.21.15.97:0/965719347 conn(0x7f0d840b01e0 msgr2=0x7f0d840b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/965719347 shutdown_connections 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.282+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/965719347 wait complete. 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 10 monclient: build_initial_monmap 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 10 monclient: monmap: 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:54.290 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 1 librados: starting msgr at 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 1 librados: starting objecter 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 1 Processor -- start 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 1 -- start start 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 1 librados: setting wanted keys 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 1 librados: calling monclient init 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.283+0000 7f0d8bc4b640 10 monclient: init 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d84072740 0x7f0d841d7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d84072740 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:54.291 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d840af360 0x7f0d841d7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d840af360 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d841dac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d84138ba0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d841cac90 con 0x7f0d84072740 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d841d7490 con 0x7f0d840af360 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d841d83a0 con 0x7f0d84138ba0 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8bc4b640 10 monclient(hunting): authenticate will time out at 2676.662500s 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8a1e8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d841dac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d899e7640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d84072740 0x7f0d841d7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d891e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d840af360 0x7f0d841d7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d899e7640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d84072740 0x7f0d841d7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45922/0 (socket says 172.21.15.97:45922) 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d8a1e8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d841dac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38822/0 (socket says 172.21.15.97:38822) 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d899e7640 1 -- 172.21.15.97:0/1635174762 learned_addr learned my addr 172.21.15.97:0/1635174762 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d899e7640 10 monclient(hunting): get_auth_request con 0x7f0d84072740 auth_method 0 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d899e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.284+0000 7f0d899e7640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d899e7640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d8a1e8640 10 monclient(hunting): get_auth_request con 0x7f0d84138ba0 auth_method 0 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d8a1e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d8a1e8640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d8a1e8640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d891e6640 10 monclient(hunting): get_auth_request con 0x7f0d840af360 auth_method 0 2023-12-07T07:55:54.293 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d891e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d891e6640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d891e6640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d8a1e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d8a1e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d8a1e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d899e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d899e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d899e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d891e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d891e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.285+0000 7f0d891e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 10 monclient(hunting): handle_auth_done global_id 17693 payload 824 2023-12-07T07:55:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 1 -- 172.21.15.97:0/1635174762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d840af360 msgr2=0x7f0d841d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d840af360 0x7f0d841d7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 1 -- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d84072740 msgr2=0x7f0d841d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d84072740 0x7f0d841d7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 10 monclient: _finish_hunting 0 2023-12-07T07:55:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 1 monclient: found mon.noname-c 2023-12-07T07:55:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:55:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.288+0000 7f0d8a1e8640 1 -- 172.21.15.97:0/1635174762 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d740094d0 con 0x7f0d84138ba0 2023-12-07T07:55:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.290+0000 7f0d8a1e8640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d841dac00 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009d90 tx=0x7f0d7c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.290+0000 7f0d7a7fc640 10 client.?.objecter ms_handle_connect 0x7f0d84138ba0 2023-12-07T07:55:54.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.290+0000 7f0d7a7fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.293+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c014480 con 0x7f0d84138ba0 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.293+0000 7f0d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.293+0000 7f0d7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.293+0000 7f0d7a7fc640 10 monclient: dump: 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.297 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 10 monclient: _finish_auth 0 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 10 monclient: _check_auth_tickets 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7c00f040 con 0x7f0d84138ba0 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 5 monclient: authenticate success, global_id 17693 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 monclient: _renew_subs 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8413f850 con 0x7f0d84138ba0 2023-12-07T07:55:54.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c014840 con 0x7f0d84138ba0 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7a7fc640 10 monclient: dump: 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.299 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d7bfff640 4 set_mon_vals no callback set 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 client.17693.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 monclient: _renew_subs 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d841db140 con 0x7f0d84138ba0 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 1 librados: init done 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 client.17693.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 15 client.17693.objecter _session_command_op_assign -1 1 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 20 client.17693.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 client.17693.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:54.300 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.294+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d8406e090 con 0x7f0d84138ba0 2023-12-07T07:55:54.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.315+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0d7c0149a0 con 0x7f0d84138ba0 2023-12-07T07:55:54.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.316+0000 7f0d7a7fc640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0d6806c850 0x7f0d6806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.316+0000 7f0d899e7640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0d6806c850 0x7f0d6806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.316+0000 7f0d899e7640 10 monclient: get_auth_request con 0x7f0d6806c850 auth_method 0 2023-12-07T07:55:54.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.316+0000 7f0d899e7640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0d6806c850 0x7f0d6806ed10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0d841dba50 tx=0x7f0d80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:54.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.316+0000 7f0d7a7fc640 10 client.17693.objecter ms_handle_connect 0x7f0d6806c850 2023-12-07T07:55:54.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f0d7c059a80 con 0x7f0d84138ba0 2023-12-07T07:55:54.321 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 10 client.17693.objecter ms_dispatch 0x7f0d84000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 3 client.17693.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 3 client.17693.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 10 client.17693.objecter checking command 1 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f0d68074180 0x7f0d68076620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 20 client.17693.objecter _get_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 20 client.17693.objecter put_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 15 client.17693.objecter _session_command_op_remove -1 1 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 20 client.17693.objecter _get_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 20 client.17693.objecter get_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 15 client.17693.objecter _session_command_op_assign 5 1 2023-12-07T07:55:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 20 client.17693.objecter put_session s=0x7f0d68073ab0 osd=5 4 2023-12-07T07:55:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 10 client.17693.objecter _send_command 1 2023-12-07T07:55:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 --> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d68076cf0 con 0x7f0d68074180 2023-12-07T07:55:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.318+0000 7f0d7a7fc640 20 client.17693.objecter dump_active .. 0 homeless 2023-12-07T07:55:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.319+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d7c055dc0 con 0x7f0d84138ba0 2023-12-07T07:55:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.319+0000 7f0d7a7fc640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.319+0000 7f0d891e6640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f0d68074180 0x7f0d68076620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.319+0000 7f0d891e6640 10 monclient: get_auth_request con 0x7f0d68074180 auth_method 0 2023-12-07T07:55:54.324 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.322+0000 7f0d891e6640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f0d68074180 0x7f0d68076620 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:54.324 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.322+0000 7f0d7a7fc640 10 client.17693.objecter ms_handle_connect 0x7f0d68074180 2023-12-07T07:55:54.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.346+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== osd.5 v2:172.21.15.154:6800/4174110211 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0d68076cf0 con 0x7f0d68074180 2023-12-07T07:55:54.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.346+0000 7f0d7a7fc640 10 client.17693.objecter ms_dispatch 0x7f0d84000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:54.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.346+0000 7f0d7a7fc640 10 client.17693.objecter _finish_command 1 = system:0 2023-12-07T07:55:54.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.346+0000 7f0d7a7fc640 20 client.17693.objecter put_session s=0x7f0d68073ab0 osd=5 4 2023-12-07T07:55:54.349 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.346+0000 7f0d7a7fc640 15 client.17693.objecter _session_command_op_remove 5 1 2023-12-07T07:55:54.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 10 client.17693.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 15 client.17693.objecter _session_command_op_assign -1 2 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 20 client.17693.objecter _get_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 20 client.17693.objecter put_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 20 client.17693.objecter _get_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 15 client.17693.objecter _session_command_op_remove -1 2 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 20 client.17693.objecter get_session s=0x7f0d68073ab0 osd=5 3 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 15 client.17693.objecter _session_command_op_assign 5 2 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 20 client.17693.objecter put_session s=0x7f0d68073ab0 osd=5 4 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 10 client.17693.objecter _send_command 2 2023-12-07T07:55:54.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.361+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 --> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0d84075af0 con 0x7f0d68074180 2023-12-07T07:55:54.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d7a7fc640 1 -- 172.21.15.97:0/1635174762 <== osd.5 v2:172.21.15.154:6800/4174110211 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0d84075af0 con 0x7f0d68074180 2023-12-07T07:55:54.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d7a7fc640 10 client.17693.objecter ms_dispatch 0x7f0d84000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:54.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d7a7fc640 10 client.17693.objecter _finish_command 2 = system:0 2023-12-07T07:55:54.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d7a7fc640 20 client.17693.objecter put_session s=0x7f0d68073ab0 osd=5 4 2023-12-07T07:55:54.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d7a7fc640 15 client.17693.objecter _session_command_op_remove 5 2 2023-12-07T07:55:54.377 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:54.377 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:54.377 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:54.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 10 librados: watch_flush enter 2023-12-07T07:55:54.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 10 librados: watch_flush exit 2023-12-07T07:55:54.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 10 client.17693.objecter close_session for osd.5 2023-12-07T07:55:54.380 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f0d68074180 msgr2=0x7f0d68076620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.154:6800/4174110211,v1:172.21.15.154:6801/4174110211] conn(0x7f0d68074180 0x7f0d68076620 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 20 client.17693.objecter put_session s=0x7f0d68073ab0 osd=5 1 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 20 client.17693.objecter shutdown clearing up homeless session... 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 10 client.17693.objecter successfully canceled tick 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0d6806c850 msgr2=0x7f0d6806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f0d6806c850 0x7f0d6806ed10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0d841dba50 tx=0x7f0d80009040 comp rx=0 tx=0).stop 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 10 monclient: shutdown 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 msgr2=0x7f0d841dac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8bc4b640 1 --2- 172.21.15.97:0/1635174762 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f0d84138ba0 0x7f0d841dac00 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009d90 tx=0x7f0d7c00c490 comp rx=0 tx=0).stop 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.374+0000 7f0d8a1e8640 1 -- 172.21.15.97:0/1635174762 reap_dead start 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.375+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 shutdown_connections 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.375+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 >> 172.21.15.97:0/1635174762 conn(0x7f0d840b01e0 msgr2=0x7f0d840ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:54.381 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.375+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 shutdown_connections 2023-12-07T07:55:54.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.375+0000 7f0d8bc4b640 1 -- 172.21.15.97:0/1635174762 wait complete. 2023-12-07T07:55:54.382 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.375+0000 7f0d8bc4b640 1 librados: shutdown 2023-12-07T07:55:54.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.487+0000 7f7f42661640 10 monclient: get_monmap_and_config 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.487+0000 7f7f42661640 10 monclient: build_initial_monmap 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.487+0000 7f7f42661640 10 monclient: monmap: 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:54.488655+0000 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:54.488655+0000 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.490 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:54.491 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:54.491 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.492 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.488+0000 7f7f42661640 1 Processor -- start 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.488+0000 7f7f42661640 1 -- start start 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.488+0000 7f7f42661640 10 monclient: init 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.488+0000 7f7f42661640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f3c0b1490 0x7f7f3c13f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): picked mon.noname-c con 0x7f7f3c0b1490 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c13f7a0 0x7f7f3c0719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): picked mon.noname-b con 0x7f7f3c13f7a0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7f3c071f10 0x7f7f3c06b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3c071f10 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c0b1870 con 0x7f7f3c071f10 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c0711a0 con 0x7f7f3c0b1490 2023-12-07T07:55:54.493 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0713d0 con 0x7f7f3c13f7a0 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f42661640 10 monclient(hunting): authenticate will time out at 2676.867400s 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f3b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c13f7a0 0x7f7f3c0719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.489+0000 7f7f3bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f3c0b1490 0x7f7f3c13f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f3c0b1490 0x7f7f3c13f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:42144/0 (socket says 172.21.15.97:42144) 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 1 -- 172.21.15.97:0/3646408276 learned_addr learned my addr 172.21.15.97:0/3646408276 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3b7fe640 10 monclient(hunting): get_auth_request con 0x7f7f3c13f7a0 auth_method 0 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 10 monclient(hunting): get_auth_request con 0x7f7f3c0b1490 auth_method 0 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.494 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.495 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.490+0000 7f7f3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.491+0000 7f7f40bfe640 1 --2- 172.21.15.97:0/3646408276 >> v2:172.21.15.97:3300/0 conn(0x7f7f3c071f10 0x7f7f3c06b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.491+0000 7f7f40bfe640 10 monclient(hunting): get_auth_request con 0x7f7f3c071f10 auth_method 0 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.491+0000 7f7f40bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.491+0000 7f7f40bfe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.491+0000 7f7f40bfe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_done global_id 20239 payload 274 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 1 -- 172.21.15.97:0/3646408276 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c13f7a0 msgr2=0x7f7f3c0719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 1 --2- 172.21.15.97:0/3646408276 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c13f7a0 0x7f7f3c0719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 1 -- 172.21.15.97:0/3646408276 >> v2:172.21.15.97:3300/0 conn(0x7f7f3c071f10 msgr2=0x7f7f3c06b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 1 --2- 172.21.15.97:0/3646408276 >> v2:172.21.15.97:3300/0 conn(0x7f7f3c071f10 0x7f7f3c06b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 1 monclient: found mon.noname-c 2023-12-07T07:55:54.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:54.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:54.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.492+0000 7f7f3bfff640 1 -- 172.21.15.97:0/3646408276 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c06d020 con 0x7f7f3c0b1490 2023-12-07T07:55:54.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.494+0000 7f7f3bfff640 1 --2- 172.21.15.97:0/3646408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c0b1490 0x7f7f3c13f260 secure :-1 s=READY pgs=4872 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c074990 tx=0x7f7f2c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bd17c16fd0e644e server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 1 -- 172.21.15.97:0/3646408276 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0317e0 con 0x7f7f3c0b1490 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: dump: 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.500 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: _finish_auth 0 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 1 -- 172.21.15.97:0/3646408276 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c031940 con 0x7f7f3c0b1490 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f42661640 5 monclient: authenticate success, global_id 20239 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 1 -- 172.21.15.97:0/3646408276 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c031cd0 con 0x7f7f3c0b1490 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:54.501 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f3affd640 10 monclient: dump: 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f42661640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:54.502 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.497+0000 7f7f42661640 4 set_mon_vals no callback set 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 10 monclient: shutdown 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 -- 172.21.15.97:0/3646408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c0b1490 msgr2=0x7f7f3c13f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 --2- 172.21.15.97:0/3646408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c0b1490 0x7f7f3c13f260 secure :-1 s=READY pgs=4872 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c074990 tx=0x7f7f2c02b880 comp rx=0 tx=0).stop 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 -- 172.21.15.97:0/3646408276 shutdown_connections 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 --2- 172.21.15.97:0/3646408276 >> v2:172.21.15.97:3300/0 conn(0x7f7f3c071f10 0x7f7f3c06b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 --2- 172.21.15.97:0/3646408276 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c13f7a0 0x7f7f3c0719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 --2- 172.21.15.97:0/3646408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c0b1490 0x7f7f3c13f260 unknown :-1 s=CLOSED pgs=4872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.505 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 -- 172.21.15.97:0/3646408276 >> 172.21.15.97:0/3646408276 conn(0x7f7f3c0b2310 msgr2=0x7f7f3c0b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 -- 172.21.15.97:0/3646408276 shutdown_connections 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.498+0000 7f7f42661640 1 -- 172.21.15.97:0/3646408276 wait complete. 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.499+0000 7f7f42661640 10 monclient: build_initial_monmap 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.499+0000 7f7f42661640 10 monclient: monmap: 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.506 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.500+0000 7f7f42661640 1 librados: starting msgr at 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.500+0000 7f7f42661640 1 librados: starting objecter 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.500+0000 7f7f42661640 1 Processor -- start 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.500+0000 7f7f42661640 1 -- start start 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.500+0000 7f7f42661640 1 librados: setting wanted keys 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.500+0000 7f7f42661640 1 librados: calling monclient init 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.500+0000 7f7f42661640 10 monclient: init 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f3c071f10 0x7f7f3c1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3c071f10 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c0b1490 0x7f7f3c134a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): picked mon.noname-c con 0x7f7f3c0b1490 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c13f7a0 0x7f7f3c134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): picked mon.noname-b con 0x7f7f3c13f7a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:54.507 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c1404f0 con 0x7f7f3c071f10 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c1354a0 con 0x7f7f3c13f7a0 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c145310 con 0x7f7f3c0b1490 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.501+0000 7f7f42661640 10 monclient(hunting): authenticate will time out at 2676.880013s 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f3c071f10 0x7f7f3c1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f40bfe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c13f7a0 0x7f7f3c134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c0b1490 0x7f7f3c134a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f3c071f10 0x7f7f3c1448d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45940/0 (socket says 172.21.15.97:45940) 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3bfff640 1 -- 172.21.15.97:0/328634031 learned_addr learned my addr 172.21.15.97:0/328634031 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3bfff640 10 monclient(hunting): get_auth_request con 0x7f7f3c071f10 auth_method 0 2023-12-07T07:55:54.508 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f40bfe640 10 monclient(hunting): get_auth_request con 0x7f7f3c13f7a0 auth_method 0 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f40bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f40bfe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f40bfe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3b7fe640 10 monclient(hunting): get_auth_request con 0x7f7f3c0b1490 auth_method 0 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.502+0000 7f7f3b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f40bfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f40bfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.509 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f40bfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.510 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:54.510 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.503+0000 7f7f3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 10 monclient(hunting): handle_auth_done global_id 16194 payload 824 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 1 -- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c0b1490 msgr2=0x7f7f3c134a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c0b1490 0x7f7f3c134a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 1 -- 172.21.15.97:0/328634031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c13f7a0 msgr2=0x7f7f3c134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c13f7a0 0x7f7f3c134f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 1 monclient: found mon.noname-a 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.505+0000 7f7f3bfff640 1 -- 172.21.15.97:0/328634031 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c0094d0 con 0x7f7f3c071f10 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.507+0000 7f7f3bfff640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f3c071f10 0x7f7f3c1448d0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0090b0 tx=0x7f7f2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.507+0000 7f7f38ff9640 10 client.?.objecter ms_handle_connect 0x7f7f3c071f10 2023-12-07T07:55:54.511 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.507+0000 7f7f38ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 1 -- 172.21.15.97:0/328634031 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c032880 con 0x7f7f3c071f10 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: dump: 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.515 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: _finish_auth 0 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 1 -- 172.21.15.97:0/328634031 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c032a00 con 0x7f7f3c071f10 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 5 monclient: authenticate success, global_id 16194 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 10 monclient: _renew_subs 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c13c630 con 0x7f7f3c071f10 2023-12-07T07:55:54.516 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 1 -- 172.21.15.97:0/328634031 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c032db0 con 0x7f7f3c071f10 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f38ff9640 10 monclient: dump: 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:54.517 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f39ffb640 4 set_mon_vals no callback set 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 10 client.16194.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 10 monclient: _renew_subs 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c140090 con 0x7f7f3c071f10 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.512+0000 7f7f42661640 1 librados: init done 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.513+0000 7f7f42661640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.513+0000 7f7f42661640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.513+0000 7f7f42661640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:54.518 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.513+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c067ac0 con 0x7f7f3c071f10 2023-12-07T07:55:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.764+0000 7f7f38ff9640 1 -- 172.21.15.97:0/328634031 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7f2c04d3f0 con 0x7f7f3c071f10 2023-12-07T07:55:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.764+0000 7f7f38ff9640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f7f2806c820 0x7f7f2806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.764+0000 7f7f3b7fe640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f7f2806c820 0x7f7f2806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.764+0000 7f7f3b7fe640 10 monclient: get_auth_request con 0x7f7f2806c820 auth_method 0 2023-12-07T07:55:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.765+0000 7f7f3b7fe640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f7f2806c820 0x7f7f2806ece0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c141740 tx=0x7f7f30009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:54.767 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.765+0000 7f7f38ff9640 10 client.16194.objecter ms_handle_connect 0x7f7f2806c820 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 1 -- 172.21.15.97:0/328634031 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f7f2c079ff0 con 0x7f7f3c071f10 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 10 client.16194.objecter ms_dispatch 0x7f7f3c000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 3 client.16194.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 3 client.16194.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 20 client.16194.objecter dump_active .. 0 homeless 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 1 -- 172.21.15.97:0/328634031 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f2c076330 con 0x7f7f3c071f10 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.778+0000 7f7f38ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:54.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.961+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f3c1402e0 con 0x7f7f2806c820 2023-12-07T07:55:54.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.962+0000 7f7f38ff9640 1 -- 172.21.15.97:0/328634031 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7f30004020 con 0x7f7f2806c820 2023-12-07T07:55:54.964 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 5 to deep-scrub 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 10 librados: watch_flush enter 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 10 librados: watch_flush exit 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 20 client.16194.objecter shutdown clearing up homeless session... 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 10 client.16194.objecter successfully canceled tick 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f7f2806c820 msgr2=0x7f7f2806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f7f2806c820 0x7f7f2806ece0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c141740 tx=0x7f7f30009190 comp rx=0 tx=0).stop 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 10 monclient: shutdown 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:54.967 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f3c071f10 msgr2=0x7f7f3c1448d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f3c071f10 0x7f7f3c1448d0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0090b0 tx=0x7f7f2c0046c0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 shutdown_connections 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f7f2806c820 0x7f7f2806ece0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c13f7a0 0x7f7f3c134f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7f3c0b1490 0x7f7f3c134a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 --2- 172.21.15.97:0/328634031 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f3c071f10 0x7f7f3c1448d0 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 >> 172.21.15.97:0/328634031 conn(0x7f7f3c0b2310 msgr2=0x7f7f3c139020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 shutdown_connections 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 -- 172.21.15.97:0/328634031 wait complete. 2023-12-07T07:55:54.968 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:54.964+0000 7f7f42661640 1 librados: shutdown 2023-12-07T07:55:54.978 INFO:tasks.ceph:Scrubbing osd.6 2023-12-07T07:55:54.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:55.083 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.081+0000 7feed4bd0640 10 monclient: get_monmap_and_config 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.081+0000 7feed4bd0640 10 monclient: build_initial_monmap 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.081+0000 7feed4bd0640 10 monclient: monmap: 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:55.082367+0000 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:55.082367+0000 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:55.084 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 1 Processor -- start 2023-12-07T07:55:55.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 1 -- start start 2023-12-07T07:55:55.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient: init 2023-12-07T07:55:55.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:55.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-12-07T07:55:55.086 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feed0145840 0x7feed0146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient(hunting): picked mon.noname-c con 0x7feed0145840 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feed00b20f0 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feed00b2520 0x7feed014c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient(hunting): picked mon.noname-a con 0x7feed00b2520 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed00aa850 con 0x7feed00b20f0 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed01465c0 con 0x7feed00b2520 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed0145c20 con 0x7feed0145840 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.082+0000 7feed4bd0640 10 monclient(hunting): authenticate will time out at 2677.460975s 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feece59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feed0145840 0x7feed0146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feeced9d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7feed00b2520 0x7feed014c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feece59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feed0145840 0x7feed0146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:42164/0 (socket says 172.21.15.97:42164) 2023-12-07T07:55:55.087 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feece59c640 1 -- 172.21.15.97:0/38450892 learned_addr learned my addr 172.21.15.97:0/38450892 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feece59c640 10 monclient(hunting): get_auth_request con 0x7feed0145840 auth_method 0 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feece59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feece59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feece59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feeced9d640 10 monclient(hunting): get_auth_request con 0x7feed00b2520 auth_method 0 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feeced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feeced9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.083+0000 7feeced9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.084+0000 7feece59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.084+0000 7feece59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.088 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.084+0000 7feece59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.084+0000 7feeced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.084+0000 7feeced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.084+0000 7feeced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 10 monclient(hunting): handle_auth_done global_id 20245 payload 274 2023-12-07T07:55:55.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 1 -- 172.21.15.97:0/38450892 >> v2:172.21.15.97:3300/0 conn(0x7feed00b2520 msgr2=0x7feed014c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:55.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 1 --2- 172.21.15.97:0/38450892 >> v2:172.21.15.97:3300/0 conn(0x7feed00b2520 0x7feed014c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 1 -- 172.21.15.97:0/38450892 >> v1:172.21.15.97:6789/0 conn(0x7feed00b20f0 legacy=0x7feed013f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 10 monclient: _finish_hunting 0 2023-12-07T07:55:55.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 1 monclient: found mon.noname-c 2023-12-07T07:55:55.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:55.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.085+0000 7feece59c640 1 -- 172.21.15.97:0/38450892 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed00a9260 con 0x7feed0145840 2023-12-07T07:55:55.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.087+0000 7feece59c640 1 --2- 172.21.15.97:0/38450892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed0145840 0x7feed0146080 secure :-1 s=READY pgs=4873 cs=0 l=1 rev1=1 crypto rx=0x7feed00b7600 tx=0x7feec002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45c11ccdfb910bc server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 1 -- 172.21.15.97:0/38450892 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0031870 con 0x7feed0145840 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: dump: 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.094 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: _finish_auth 0 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: _check_auth_tickets 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 1 -- 172.21.15.97:0/38450892 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec00319d0 con 0x7feed0145840 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 1 -- 172.21.15.97:0/38450892 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0031d60 con 0x7feed0145840 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feecdd9b640 10 monclient: dump: 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.095 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feed4bd0640 5 monclient: authenticate success, global_id 20245 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feed4bd0640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:55.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.091+0000 7feed4bd0640 4 set_mon_vals no callback set 2023-12-07T07:55:55.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 10 monclient: shutdown 2023-12-07T07:55:55.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:55.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 -- 172.21.15.97:0/38450892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed0145840 msgr2=0x7feed0146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 --2- 172.21.15.97:0/38450892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed0145840 0x7feed0146080 secure :-1 s=READY pgs=4873 cs=0 l=1 rev1=1 crypto rx=0x7feed00b7600 tx=0x7feec002b910 comp rx=0 tx=0).stop 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 -- 172.21.15.97:0/38450892 shutdown_connections 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 --2- 172.21.15.97:0/38450892 >> v2:172.21.15.97:3300/0 conn(0x7feed00b2520 0x7feed014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 --2- 172.21.15.97:0/38450892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed0145840 0x7feed0146080 unknown :-1 s=CLOSED pgs=4873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 -- 172.21.15.97:0/38450892 >> 172.21.15.97:0/38450892 conn(0x7feed00b41f0 msgr2=0x7feed00b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 -- 172.21.15.97:0/38450892 shutdown_connections 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.092+0000 7feed4bd0640 1 -- 172.21.15.97:0/38450892 wait complete. 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 10 monclient: build_initial_monmap 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 10 monclient: monmap: 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:55.102 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 1 librados: starting msgr at 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 1 librados: starting objecter 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 1 Processor -- start 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 1 -- start start 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 1 librados: setting wanted keys 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 1 librados: calling monclient init 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.093+0000 7feed4bd0640 10 monclient: init 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feed00b20f0 0x7feed00aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): picked mon.noname-c con 0x7feed00b20f0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:55.103 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed00b2520 0x7feed01d7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): picked mon.noname-b con 0x7feed00b2520 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed0145840 0x7feed00afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): picked mon.noname-a con 0x7feed0145840 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed01d8290 con 0x7feed0145840 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed00ac7c0 con 0x7feed00b2520 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed01d7870 con 0x7feed00b20f0 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:55.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feed4bd0640 10 monclient(hunting): authenticate will time out at 2677.472373s 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feeced9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed0145840 0x7feed00afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feece59c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feed00b20f0 0x7feed00aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feec7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed00b2520 0x7feed01d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feeced9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed0145840 0x7feed00afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45956/0 (socket says 172.21.15.97:45956) 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feeced9d640 1 -- 172.21.15.97:0/2324206958 learned_addr learned my addr 172.21.15.97:0/2324206958 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feeced9d640 10 monclient(hunting): get_auth_request con 0x7feed0145840 auth_method 0 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feeced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feeced9d640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feeced9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feece59c640 10 monclient(hunting): get_auth_request con 0x7feed00b20f0 auth_method 0 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feece59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feece59c640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feece59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feec7fff640 10 monclient(hunting): get_auth_request con 0x7feed00b2520 auth_method 0 2023-12-07T07:55:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feec7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.094+0000 7feec7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feeced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feeced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feeced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feece59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feece59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.095+0000 7feece59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.097+0000 7feeced9d640 10 monclient(hunting): handle_auth_done global_id 16203 payload 824 2023-12-07T07:55:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 1 -- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feed00b20f0 msgr2=0x7feed00aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feed00b20f0 0x7feed00aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 1 -- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed00b2520 msgr2=0x7feed01d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed00b2520 0x7feed01d7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 10 monclient: _finish_hunting 0 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 1 monclient: found mon.noname-a 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feeced9d640 1 -- 172.21.15.97:0/2324206958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec00094d0 con 0x7feed0145840 2023-12-07T07:55:55.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.098+0000 7feec7fff640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed00b2520 0x7feed01d7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-07T07:55:55.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.100+0000 7feeced9d640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed0145840 0x7feed00afa30 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7feec800efc0 tx=0x7feec800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.100+0000 7feec6ffd640 10 client.?.objecter ms_handle_connect 0x7feed0145840 2023-12-07T07:55:55.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.100+0000 7feec6ffd640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:55.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8014480 con 0x7feed0145840 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: dump: 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: _finish_auth 0 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec800f040 con 0x7feed0145840 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8014820 con 0x7feed0145840 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 5 monclient: authenticate success, global_id 16203 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 10 monclient: _renew_subs 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed00ae360 con 0x7feed0145840 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feec6ffd640 10 monclient: dump: 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.110 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feecd59a640 4 set_mon_vals no callback set 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 10 client.16203.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 10 monclient: _renew_subs 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed00b0150 con 0x7feed0145840 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.103+0000 7feed4bd0640 1 librados: init done 2023-12-07T07:55:55.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.104+0000 7feed4bd0640 10 client.16203.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:55.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.104+0000 7feed4bd0640 15 client.16203.objecter _session_command_op_assign -1 1 2023-12-07T07:55:55.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.104+0000 7feed4bd0640 20 client.16203.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:55.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.104+0000 7feed4bd0640 10 client.16203.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:55.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.104+0000 7feed4bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:55.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.104+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feed00ae640 con 0x7feed0145840 2023-12-07T07:55:55.125 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.122+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feec8014980 con 0x7feed0145840 2023-12-07T07:55:55.125 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.122+0000 7feec6ffd640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feeb006c820 0x7feeb006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.125 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.123+0000 7feece59c640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feeb006c820 0x7feeb006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.125 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.123+0000 7feece59c640 10 monclient: get_auth_request con 0x7feeb006c820 auth_method 0 2023-12-07T07:55:55.126 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.123+0000 7feece59c640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feeb006c820 0x7feeb006ece0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feed00b7600 tx=0x7feec003c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.126 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.123+0000 7feec6ffd640 10 client.16203.objecter ms_handle_connect 0x7feeb006c820 2023-12-07T07:55:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7feec8059ac0 con 0x7feed0145840 2023-12-07T07:55:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 10 client.16203.objecter ms_dispatch 0x7feed0000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 3 client.16203.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 3 client.16203.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 10 client.16203.objecter checking command 1 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7feeb0074150 0x7feeb00765f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 20 client.16203.objecter _get_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 20 client.16203.objecter put_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 15 client.16203.objecter _session_command_op_remove -1 1 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 20 client.16203.objecter _get_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 20 client.16203.objecter get_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 15 client.16203.objecter _session_command_op_assign 6 1 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 20 client.16203.objecter put_session s=0x7feeb0073a80 osd=6 4 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 10 client.16203.objecter _send_command 1 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 --> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feeb0076cc0 con 0x7feeb0074150 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.125+0000 7feec6ffd640 20 client.16203.objecter dump_active .. 0 homeless 2023-12-07T07:55:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.126+0000 7feec7fff640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7feeb0074150 0x7feeb00765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.126+0000 7feec7fff640 10 monclient: get_auth_request con 0x7feeb0074150 auth_method 0 2023-12-07T07:55:55.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.127+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7feec8055e00 con 0x7feed0145840 2023-12-07T07:55:55.130 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.127+0000 7feec6ffd640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:55.131 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.129+0000 7feec7fff640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7feeb0074150 0x7feeb00765f0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.131 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.129+0000 7feec6ffd640 10 client.16203.objecter ms_handle_connect 0x7feeb0074150 2023-12-07T07:55:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.153+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== osd.6 v2:172.21.15.154:6824/1615374996 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7feeb0076cc0 con 0x7feeb0074150 2023-12-07T07:55:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.153+0000 7feec6ffd640 10 client.16203.objecter ms_dispatch 0x7feed0000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.153+0000 7feec6ffd640 10 client.16203.objecter _finish_command 1 = system:0 2023-12-07T07:55:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.153+0000 7feec6ffd640 20 client.16203.objecter put_session s=0x7feeb0073a80 osd=6 4 2023-12-07T07:55:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.153+0000 7feec6ffd640 15 client.16203.objecter _session_command_op_remove 6 1 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 10 client.16203.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 15 client.16203.objecter _session_command_op_assign -1 2 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 20 client.16203.objecter _get_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 20 client.16203.objecter put_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 20 client.16203.objecter _get_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 15 client.16203.objecter _session_command_op_remove -1 2 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 20 client.16203.objecter get_session s=0x7feeb0073a80 osd=6 3 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 15 client.16203.objecter _session_command_op_assign 6 2 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 20 client.16203.objecter put_session s=0x7feeb0073a80 osd=6 4 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 10 client.16203.objecter _send_command 2 2023-12-07T07:55:55.168 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.165+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 --> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feed00aad80 con 0x7feeb0074150 2023-12-07T07:55:55.180 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feec6ffd640 1 -- 172.21.15.97:0/2324206958 <== osd.6 v2:172.21.15.154:6824/1615374996 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7feeb8001810 con 0x7feeb0074150 2023-12-07T07:55:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feec6ffd640 10 client.16203.objecter ms_dispatch 0x7feed0000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feec6ffd640 10 client.16203.objecter _finish_command 2 = system:0 2023-12-07T07:55:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feec6ffd640 20 client.16203.objecter put_session s=0x7feeb0073a80 osd=6 4 2023-12-07T07:55:55.181 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feec6ffd640 15 client.16203.objecter _session_command_op_remove 6 2 2023-12-07T07:55:55.181 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:55.181 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:55.182 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:55.183 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 10 librados: watch_flush enter 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 10 librados: watch_flush exit 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 10 client.16203.objecter close_session for osd.6 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7feeb0074150 msgr2=0x7feeb00765f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.154:6824/1615374996,v1:172.21.15.154:6825/1615374996] conn(0x7feeb0074150 0x7feeb00765f0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 20 client.16203.objecter put_session s=0x7feeb0073a80 osd=6 1 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 20 client.16203.objecter shutdown clearing up homeless session... 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 10 client.16203.objecter successfully canceled tick 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feeb006c820 msgr2=0x7feeb006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7feeb006c820 0x7feeb006ece0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feed00b7600 tx=0x7feec003c040 comp rx=0 tx=0).stop 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 10 monclient: shutdown 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed0145840 msgr2=0x7feed00afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 --2- 172.21.15.97:0/2324206958 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feed0145840 0x7feed00afa30 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7feec800efc0 tx=0x7feec800c490 comp rx=0 tx=0).stop 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feeced9d640 1 -- 172.21.15.97:0/2324206958 reap_dead start 2023-12-07T07:55:55.184 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 shutdown_connections 2023-12-07T07:55:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 >> 172.21.15.97:0/2324206958 conn(0x7feed00b41f0 msgr2=0x7feed00b7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.178+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 shutdown_connections 2023-12-07T07:55:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.179+0000 7feed4bd0640 1 -- 172.21.15.97:0/2324206958 wait complete. 2023-12-07T07:55:55.185 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.179+0000 7feed4bd0640 1 librados: shutdown 2023-12-07T07:55:55.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.289+0000 7fb1a2246640 10 monclient: get_monmap_and_config 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.289+0000 7fb1a2246640 10 monclient: build_initial_monmap 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.289+0000 7fb1a2246640 10 monclient: monmap: 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:55.290688+0000 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:55.290688+0000 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:55.293 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:55.293 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:55.293 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:55.293 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.290+0000 7fb1a2246640 1 Processor -- start 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.290+0000 7fb1a2246640 1 -- start start 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.290+0000 7fb1a2246640 10 monclient: init 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb19c0a8a30 0x7fb19c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c0a8a30 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb19c0a8e10 0x7fb19c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient(hunting): picked mon.noname-c con 0x7fb19c0a8e10 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb19c1454a0 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c0aac10 con 0x7fb19c1454a0 2023-12-07T07:55:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19c0aaa10 con 0x7fb19c0a8a30 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19c0aaef0 con 0x7fb19c0a8e10 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb1a2246640 10 monclient(hunting): authenticate will time out at 2677.669431s 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.291+0000 7fb192ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb19c0a8e10 0x7fb19c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb19c0a8e10 0x7fb19c142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:42180/0 (socket says 172.21.15.97:42180) 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 1 -- 172.21.15.97:0/4212978826 learned_addr learned my addr 172.21.15.97:0/4212978826 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 10 monclient(hunting): get_auth_request con 0x7fb19c0a8e10 auth_method 0 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.296 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.292+0000 7fb192ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 10 monclient(hunting): handle_auth_done global_id 20251 payload 274 2023-12-07T07:55:55.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 1 -- 172.21.15.97:0/4212978826 >> v2:172.21.15.97:3300/0 conn(0x7fb19c0a8a30 msgr2=0x7fb19c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:55.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 1 --2- 172.21.15.97:0/4212978826 >> v2:172.21.15.97:3300/0 conn(0x7fb19c0a8a30 0x7fb19c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 1 -- 172.21.15.97:0/4212978826 >> v1:172.21.15.154:6789/0 conn(0x7fb19c1454a0 legacy=0x7fb19c1458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:55.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 10 monclient: _finish_hunting 0 2023-12-07T07:55:55.297 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 1 monclient: found mon.noname-c 2023-12-07T07:55:55.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:55.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.298 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.294+0000 7fb192ffd640 1 -- 172.21.15.97:0/4212978826 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0a97f0 con 0x7fb19c0a8e10 2023-12-07T07:55:55.299 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.296+0000 7fb192ffd640 1 --2- 172.21.15.97:0/4212978826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c0a8e10 0x7fb19c142a20 secure :-1 s=READY pgs=4874 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0b87d0 tx=0x7fb18402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=618c20be67445d39 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 1 -- 172.21.15.97:0/4212978826 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb184030860 con 0x7fb19c0a8e10 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: dump: 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.303 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: _finish_auth 0 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: _check_auth_tickets 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 1 -- 172.21.15.97:0/4212978826 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1840309c0 con 0x7fb19c0a8e10 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 1 -- 172.21.15.97:0/4212978826 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb184030d50 con 0x7fb19c0a8e10 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb1a2246640 5 monclient: authenticate success, global_id 20251 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb1a2246640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb1a2246640 4 set_mon_vals no callback set 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.300+0000 7fb19affd640 10 monclient: dump: 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.305 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 10 monclient: shutdown 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 -- 172.21.15.97:0/4212978826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c0a8e10 msgr2=0x7fb19c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 --2- 172.21.15.97:0/4212978826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c0a8e10 0x7fb19c142a20 secure :-1 s=READY pgs=4874 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0b87d0 tx=0x7fb18402b9a0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 -- 172.21.15.97:0/4212978826 shutdown_connections 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 --2- 172.21.15.97:0/4212978826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c0a8e10 0x7fb19c142a20 unknown :-1 s=CLOSED pgs=4874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 --2- 172.21.15.97:0/4212978826 >> v2:172.21.15.97:3300/0 conn(0x7fb19c0a8a30 0x7fb19c1424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.308 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 -- 172.21.15.97:0/4212978826 >> 172.21.15.97:0/4212978826 conn(0x7fb19c061a30 msgr2=0x7fb19c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 -- 172.21.15.97:0/4212978826 shutdown_connections 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.301+0000 7fb1a2246640 1 -- 172.21.15.97:0/4212978826 wait complete. 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.302+0000 7fb1a2246640 10 monclient: build_initial_monmap 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.302+0000 7fb1a2246640 10 monclient: monmap: 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:55.309 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.302+0000 7fb1a2246640 1 librados: starting msgr at 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.302+0000 7fb1a2246640 1 librados: starting objecter 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.302+0000 7fb1a2246640 1 Processor -- start 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 -- start start 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 librados: setting wanted keys 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 librados: calling monclient init 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient: init 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19c0a8a30 0x7fb19c0aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c0a8a30 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb19c0a8e10 0x7fb19c1d73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.310 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): picked mon.noname-c con 0x7fb19c0a8e10 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c1454a0 0x7fb19c13b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): picked mon.noname-b con 0x7fb19c1454a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c0ac250 con 0x7fb19c0a8a30 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c0af1b0 con 0x7fb19c1454a0 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c0adfe0 con 0x7fb19c0a8e10 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.303+0000 7fb1a2246640 10 monclient(hunting): authenticate will time out at 2677.681863s 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb19c0a8e10 0x7fb19c1d73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c1454a0 0x7fb19c13b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19b7fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19c0a8a30 0x7fb19c0aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb19c0a8e10 0x7fb19c1d73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38878/0 (socket says 172.21.15.97:38878) 2023-12-07T07:55:55.311 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 1 -- 172.21.15.97:0/1523365171 learned_addr learned my addr 172.21.15.97:0/1523365171 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 10 monclient(hunting): get_auth_request con 0x7fb19c0a8e10 auth_method 0 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 10 monclient(hunting): get_auth_request con 0x7fb19c1454a0 auth_method 0 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19b7fe640 10 monclient(hunting): get_auth_request con 0x7fb19c0a8a30 auth_method 0 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb19bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.312 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.304+0000 7fb192ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.305+0000 7fb19b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.305+0000 7fb19b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.313 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.305+0000 7fb19b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 10 monclient(hunting): handle_auth_done global_id 20254 payload 824 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 1 -- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb19c0a8e10 msgr2=0x7fb19c1d73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb19c0a8e10 0x7fb19c1d73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 1 -- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19c0a8a30 msgr2=0x7fb19c0aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19c0a8a30 0x7fb19c0aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 10 monclient: _finish_hunting 0 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 1 monclient: found mon.noname-b 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.307+0000 7fb19bfff640 1 -- 172.21.15.97:0/1523365171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1840094d0 con 0x7fb19c1454a0 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.309+0000 7fb19bfff640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c1454a0 0x7fb19c13b910 secure :-1 s=READY pgs=4875 cs=0 l=1 rev1=1 crypto rx=0x7fb19400be20 tx=0x7fb194009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.309+0000 7fb198ff9640 10 client.?.objecter ms_handle_connect 0x7fb19c1454a0 2023-12-07T07:55:55.314 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.309+0000 7fb198ff9640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:55.315 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 1 -- 172.21.15.97:0/1523365171 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19400cdb0 con 0x7fb19c1454a0 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: dump: 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: _finish_auth 0 2023-12-07T07:55:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: _check_auth_tickets 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 5 monclient: authenticate success, global_id 20254 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 1 -- 172.21.15.97:0/1523365171 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb194014440 con 0x7fb19c1454a0 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 1 -- 172.21.15.97:0/1523365171 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb194014830 con 0x7fb19c1454a0 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 10 monclient: _renew_subs 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19c1d7de0 con 0x7fb19c1454a0 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb19a7fc640 4 set_mon_vals no callback set 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb198ff9640 10 monclient: dump: 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.317 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 10 client.20254.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 10 monclient: _renew_subs 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19c13c030 con 0x7fb19c1454a0 2023-12-07T07:55:55.318 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.313+0000 7fb1a2246640 1 librados: init done 2023-12-07T07:55:55.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.314+0000 7fb1a2246640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:55.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.314+0000 7fb1a2246640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:55.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.314+0000 7fb1a2246640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.319 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.314+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19c067ac0 con 0x7fb19c1454a0 2023-12-07T07:55:55.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.561+0000 7fb198ff9640 1 -- 172.21.15.97:0/1523365171 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb19401d3b0 con 0x7fb19c1454a0 2023-12-07T07:55:55.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.561+0000 7fb198ff9640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb17c06c7d0 0x7fb17c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.561+0000 7fb19b7fe640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb17c06c7d0 0x7fb17c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.561+0000 7fb19b7fe640 10 monclient: get_auth_request con 0x7fb17c06c7d0 auth_method 0 2023-12-07T07:55:55.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.562+0000 7fb19b7fe640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb17c06c7d0 0x7fb17c06ec90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009990 tx=0x7fb18c009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.562+0000 7fb198ff9640 10 client.20254.objecter ms_handle_connect 0x7fb17c06c7d0 2023-12-07T07:55:55.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.564+0000 7fb198ff9640 1 -- 172.21.15.97:0/1523365171 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fb194058900 con 0x7fb19c1454a0 2023-12-07T07:55:55.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.564+0000 7fb198ff9640 10 client.20254.objecter ms_dispatch 0x7fb19c000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:55.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.564+0000 7fb198ff9640 3 client.20254.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:55.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.564+0000 7fb198ff9640 3 client.20254.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:55.567 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.564+0000 7fb198ff9640 20 client.20254.objecter dump_active .. 0 homeless 2023-12-07T07:55:55.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.574+0000 7fb198ff9640 1 -- 172.21.15.97:0/1523365171 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb194054c40 con 0x7fb19c1454a0 2023-12-07T07:55:55.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.574+0000 7fb198ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:55.576 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.574+0000 7fb198ff9640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:55.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.742+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fb19c146000 con 0x7fb17c06c7d0 2023-12-07T07:55:55.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.742+0000 7fb198ff9640 1 -- 172.21.15.97:0/1523365171 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb18c00e850 con 0x7fb17c06c7d0 2023-12-07T07:55:55.745 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 6 to deep-scrub 2023-12-07T07:55:55.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 10 librados: watch_flush enter 2023-12-07T07:55:55.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 10 librados: watch_flush exit 2023-12-07T07:55:55.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 20 client.20254.objecter shutdown clearing up homeless session... 2023-12-07T07:55:55.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 10 client.20254.objecter successfully canceled tick 2023-12-07T07:55:55.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb17c06c7d0 msgr2=0x7fb17c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb17c06c7d0 0x7fb17c06ec90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009990 tx=0x7fb18c009110 comp rx=0 tx=0).stop 2023-12-07T07:55:55.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 10 monclient: shutdown 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c1454a0 msgr2=0x7fb19c13b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c1454a0 0x7fb19c13b910 secure :-1 s=READY pgs=4875 cs=0 l=1 rev1=1 crypto rx=0x7fb19400be20 tx=0x7fb194009ef0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 shutdown_connections 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fb17c06c7d0 0x7fb17c06ec90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb19c1454a0 0x7fb19c13b910 unknown :-1 s=CLOSED pgs=4875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb19c0a8e10 0x7fb19c1d73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 --2- 172.21.15.97:0/1523365171 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb19c0a8a30 0x7fb19c0aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 >> 172.21.15.97:0/1523365171 conn(0x7fb19c061a30 msgr2=0x7fb19c1458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 shutdown_connections 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 -- 172.21.15.97:0/1523365171 wait complete. 2023-12-07T07:55:55.748 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.744+0000 7fb1a2246640 1 librados: shutdown 2023-12-07T07:55:55.757 INFO:tasks.ceph:Scrubbing osd.7 2023-12-07T07:55:55.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-07T07:55:55.865 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.863+0000 7fcfb3b05640 10 monclient: get_monmap_and_config 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.863+0000 7fcfb3b05640 10 monclient: build_initial_monmap 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.863+0000 7fcfb3b05640 10 monclient: monmap: 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:55.864407+0000 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:55.864407+0000 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:55.866 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:55.867 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.863+0000 7fcfb3b05640 1 Processor -- start 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 1 -- start start 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient: init 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcfac0a87c0 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac0a8ba0 0x7fcfac0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient(hunting): picked mon.noname-b con 0x7fcfac0a8ba0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfac13ef70 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac0b1980 con 0x7fcfac0a87c0 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac0aa810 con 0x7fcfac13ef70 2023-12-07T07:55:55.868 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac142c90 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:55.869 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.864+0000 7fcfb3b05640 10 monclient(hunting): authenticate will time out at 2678.243138s 2023-12-07T07:55:55.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.872+0000 7fcfb20a2640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fcfac13ef70 0x7fcfac13f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:53880/0 (socket says 172.21.15.97:53880) 2023-12-07T07:55:55.875 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.872+0000 7fcfb20a2640 1 -- 172.21.15.97:0/1467061441 learned_addr learned my addr 172.21.15.97:0/1467061441 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:55.877 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.874+0000 7fcfb10a0640 1 --2- 172.21.15.97:0/1467061441 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac0a8ba0 0x7fcfac0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb089f640 1 -- 172.21.15.97:0/1467061441 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520144462 0 0) 0x7fcfac0b1980 con 0x7fcfac0a87c0 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb089f640 10 monclient(hunting): _negotiate 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb089f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb089f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb089f640 10 monclient(hunting): my global_id is 16209 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb089f640 1 -- 172.21.15.97:0/1467061441 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf98003520 con 0x7fcfac0a87c0 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb10a0640 10 monclient(hunting): get_auth_request con 0x7fcfac0a8ba0 auth_method 0 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb10a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb10a0640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.875+0000 7fcfb10a0640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.876+0000 7fcfb089f640 1 -- 172.21.15.97:0/1467061441 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9889039 0 0) 0x7fcfac0aa810 con 0x7fcfac13ef70 2023-12-07T07:55:55.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.876+0000 7fcfb089f640 10 monclient(hunting): _negotiate 2023-12-07T07:55:55.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.876+0000 7fcfb089f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.876+0000 7fcfb089f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.876+0000 7fcfb089f640 10 monclient(hunting): my global_id is 20257 2023-12-07T07:55:55.879 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.876+0000 7fcfb089f640 1 -- 172.21.15.97:0/1467061441 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfac0b1980 con 0x7fcfac13ef70 2023-12-07T07:55:55.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.877+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.877+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.880 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.877+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_done global_id 17708 payload 274 2023-12-07T07:55:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 1 -- 172.21.15.97:0/1467061441 >> v1:172.21.15.154:6789/0 conn(0x7fcfac13ef70 legacy=0x7fcfac13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 1 -- 172.21.15.97:0/1467061441 >> v1:172.21.15.97:6789/0 conn(0x7fcfac0a87c0 legacy=0x7fcfac142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 10 monclient: _finish_hunting 0 2023-12-07T07:55:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 1 monclient: found mon.noname-b 2023-12-07T07:55:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:55:55.882 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.879+0000 7fcfb10a0640 1 -- 172.21.15.97:0/1467061441 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac0aaa80 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.883 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.880+0000 7fcfb10a0640 1 --2- 172.21.15.97:0/1467061441 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac0a8ba0 0x7fcfac0b1fa0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0097d0 tx=0x7fcf9c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d92b8f9e91190e91 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 1 -- 172.21.15.97:0/1467061441 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c030880 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: dump: 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.887 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: _finish_auth 0 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: _check_auth_tickets 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 1 -- 172.21.15.97:0/1467061441 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c0309e0 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 1 -- 172.21.15.97:0/1467061441 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c030d70 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb3b05640 5 monclient: authenticate success, global_id 17708 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb3b05640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb3b05640 4 set_mon_vals no callback set 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:55.888 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.884+0000 7fcfb089f640 10 monclient: dump: 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.889 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.892 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 10 monclient: shutdown 2023-12-07T07:55:55.892 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:55.892 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 1 -- 172.21.15.97:0/1467061441 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac0a8ba0 msgr2=0x7fcfac0b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.892 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 1 --2- 172.21.15.97:0/1467061441 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac0a8ba0 0x7fcfac0b1fa0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0097d0 tx=0x7fcf9c02b9c0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.892 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 1 -- 172.21.15.97:0/1467061441 shutdown_connections 2023-12-07T07:55:55.892 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 1 --2- 172.21.15.97:0/1467061441 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac0a8ba0 0x7fcfac0b1fa0 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 1 -- 172.21.15.97:0/1467061441 >> 172.21.15.97:0/1467061441 conn(0x7fcfac061740 msgr2=0x7fcfac0b40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 1 -- 172.21.15.97:0/1467061441 shutdown_connections 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.885+0000 7fcfb3b05640 1 -- 172.21.15.97:0/1467061441 wait complete. 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 10 monclient: build_initial_monmap 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 10 monclient: monmap: 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.893 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 1 librados: starting msgr at 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 1 librados: starting objecter 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 1 Processor -- start 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 1 -- start start 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 1 librados: setting wanted keys 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 1 librados: calling monclient init 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.886+0000 7fcfb3b05640 10 monclient: init 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfac0a87c0 0x7fcfac144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): picked mon.noname-a con 0x7fcfac0a87c0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfac0a8ba0 0x7fcfac1376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): picked mon.noname-b con 0x7fcfac0a8ba0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac13ef70 0x7fcfac0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): picked mon.noname-c con 0x7fcfac13ef70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:55.894 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac136df0 con 0x7fcfac0a87c0 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac137c10 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac136fd0 con 0x7fcfac13ef70 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb3b05640 10 monclient(hunting): authenticate will time out at 2678.265570s 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb18a1640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfac0a87c0 0x7fcfac144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb20a2640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac13ef70 0x7fcfac0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb10a0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfac0a8ba0 0x7fcfac1376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb18a1640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfac0a87c0 0x7fcfac144fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45996/0 (socket says 172.21.15.97:45996) 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb18a1640 1 -- 172.21.15.97:0/490142146 learned_addr learned my addr 172.21.15.97:0/490142146 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb18a1640 10 monclient(hunting): get_auth_request con 0x7fcfac0a87c0 auth_method 0 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb18a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.895 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb18a1640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.887+0000 7fcfb18a1640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb20a2640 10 monclient(hunting): get_auth_request con 0x7fcfac13ef70 auth_method 0 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb20a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb20a2640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb20a2640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb10a0640 10 monclient(hunting): get_auth_request con 0x7fcfac0a8ba0 auth_method 0 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb10a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb10a0640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb10a0640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb18a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb18a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb18a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb20a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb20a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb20a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.896 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:55.897 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:55.897 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.888+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:55.897 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 10 monclient(hunting): handle_auth_done global_id 20260 payload 824 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 1 -- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac13ef70 msgr2=0x7fcfac0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfac13ef70 0x7fcfac0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 1 -- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfac0a87c0 msgr2=0x7fcfac144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfac0a87c0 0x7fcfac144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 10 monclient: _finish_hunting 0 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 1 monclient: found mon.noname-b 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.891+0000 7fcfb10a0640 1 -- 172.21.15.97:0/490142146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c009400 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.893+0000 7fcfb10a0640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfac0a8ba0 0x7fcfac1376d0 secure :-1 s=READY pgs=4877 cs=0 l=1 rev1=1 crypto rx=0x7fcfac0b4e30 tx=0x7fcf9c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.893+0000 7fcfa27fc640 10 client.?.objecter ms_handle_connect 0x7fcfac0a8ba0 2023-12-07T07:55:55.898 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.893+0000 7fcfa27fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c03e040 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: dump: 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.899 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: _finish_auth 0 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: _check_auth_tickets 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c039d10 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c043680 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa27fc640 10 monclient: dump: 2023-12-07T07:55:55.900 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfb3b05640 5 monclient: authenticate success, global_id 20260 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfb3b05640 10 monclient: _renew_subs 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfb3b05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfa3fff640 4 set_mon_vals no callback set 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.896+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac1372b0 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 10 client.20260.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:55.901 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 10 monclient: _renew_subs 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac145b30 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 1 librados: init done 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 10 client.20260.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 15 client.20260.objecter _session_command_op_assign -1 1 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 20 client.20260.objecter _get_session osd=-1 returning homeless 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 10 client.20260.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:55.902 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.897+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcfac0b1bd0 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.916+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcf9c0437e0 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.916+0000 7fcfa27fc640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fcf9006c820 0x7fcf9006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.916+0000 7fcfb18a1640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fcf9006c820 0x7fcf9006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.916+0000 7fcfb18a1640 10 monclient: get_auth_request con 0x7fcf9006c820 auth_method 0 2023-12-07T07:55:55.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.916+0000 7fcfa27fc640 10 client.20260.objecter ms_handle_connect 0x7fcf9006c820 2023-12-07T07:55:55.919 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.916+0000 7fcfb18a1640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fcf9006c820 0x7fcf9006ece0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8007fd0 tx=0x7fcfa8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fcf9c07aa60 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 10 client.20260.objecter ms_dispatch 0x7fcfac000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:55.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 3 client.20260.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:55.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 3 client.20260.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:55.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 10 client.20260.objecter checking command 1 2023-12-07T07:55:55.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7fcf90074150 0x7fcf900765f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:55.921 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 20 client.20260.objecter _get_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 20 client.20260.objecter put_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 15 client.20260.objecter _session_command_op_remove -1 1 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 20 client.20260.objecter _get_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 20 client.20260.objecter get_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 15 client.20260.objecter _session_command_op_assign 7 1 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 20 client.20260.objecter put_session s=0x7fcf90073a80 osd=7 4 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 10 client.20260.objecter _send_command 1 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 --> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf90076cc0 con 0x7fcf90074150 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 20 client.20260.objecter dump_active .. 0 homeless 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=920) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf9c076da0 con 0x7fcfac0a8ba0 2023-12-07T07:55:55.922 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.918+0000 7fcfa27fc640 10 monclient: handle_get_version_reply finishing 1 version 920 2023-12-07T07:55:55.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.919+0000 7fcfb20a2640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7fcf90074150 0x7fcf900765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:55.923 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.919+0000 7fcfb20a2640 10 monclient: get_auth_request con 0x7fcf90074150 auth_method 0 2023-12-07T07:55:55.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.922+0000 7fcfb20a2640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7fcf90074150 0x7fcf900765f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:55.925 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.922+0000 7fcfa27fc640 10 client.20260.objecter ms_handle_connect 0x7fcf90074150 2023-12-07T07:55:55.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.946+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== osd.7 v2:172.21.15.154:6816/1161771449 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcf90076cc0 con 0x7fcf90074150 2023-12-07T07:55:55.948 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.946+0000 7fcfa27fc640 10 client.20260.objecter ms_dispatch 0x7fcfac000b90 command_reply(tid 1: 0 ) v1 2023-12-07T07:55:55.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.946+0000 7fcfa27fc640 10 client.20260.objecter _finish_command 1 = system:0 2023-12-07T07:55:55.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.946+0000 7fcfa27fc640 20 client.20260.objecter put_session s=0x7fcf90073a80 osd=7 4 2023-12-07T07:55:55.949 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.946+0000 7fcfa27fc640 15 client.20260.objecter _session_command_op_remove 7 1 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 10 client.20260.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 15 client.20260.objecter _session_command_op_assign -1 2 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 20 client.20260.objecter _get_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 20 client.20260.objecter put_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 20 client.20260.objecter _get_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 15 client.20260.objecter _session_command_op_remove -1 2 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 20 client.20260.objecter get_session s=0x7fcf90073a80 osd=7 3 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 15 client.20260.objecter _session_command_op_assign 7 2 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 20 client.20260.objecter put_session s=0x7fcf90073a80 osd=7 4 2023-12-07T07:55:55.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 10 client.20260.objecter _send_command 2 2023-12-07T07:55:55.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.958+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 --> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcfac075af0 con 0x7fcf90074150 2023-12-07T07:55:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfa27fc640 1 -- 172.21.15.97:0/490142146 <== osd.7 v2:172.21.15.154:6816/1161771449 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcfac075af0 con 0x7fcf90074150 2023-12-07T07:55:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfa27fc640 10 client.20260.objecter ms_dispatch 0x7fcfac000b90 command_reply(tid 2: 0 ) v1 2023-12-07T07:55:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfa27fc640 10 client.20260.objecter _finish_command 2 = system:0 2023-12-07T07:55:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfa27fc640 20 client.20260.objecter put_session s=0x7fcf90073a80 osd=7 4 2023-12-07T07:55:55.974 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfa27fc640 15 client.20260.objecter _session_command_op_remove 7 2 2023-12-07T07:55:55.975 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2023-12-07T07:55:55.975 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-07T07:55:55.975 INFO:teuthology.orchestra.run.smithi097.stdout:} 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 10 librados: watch_flush enter 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 10 librados: watch_flush exit 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 10 client.20260.objecter close_session for osd.7 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7fcf90074150 msgr2=0x7fcf900765f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:6816/1161771449,v1:172.21.15.154:6817/1161771449] conn(0x7fcf90074150 0x7fcf900765f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 20 client.20260.objecter put_session s=0x7fcf90073a80 osd=7 1 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 20 client.20260.objecter shutdown clearing up homeless session... 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 10 client.20260.objecter successfully canceled tick 2023-12-07T07:55:55.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fcf9006c820 msgr2=0x7fcf9006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fcf9006c820 0x7fcf9006ece0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8007fd0 tx=0x7fcfa8009040 comp rx=0 tx=0).stop 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 10 monclient: shutdown 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfac0a8ba0 msgr2=0x7fcfac1376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 --2- 172.21.15.97:0/490142146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfac0a8ba0 0x7fcfac1376d0 secure :-1 s=READY pgs=4877 cs=0 l=1 rev1=1 crypto rx=0x7fcfac0b4e30 tx=0x7fcf9c004730 comp rx=0 tx=0).stop 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb20a2640 1 -- 172.21.15.97:0/490142146 reap_dead start 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 shutdown_connections 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 >> 172.21.15.97:0/490142146 conn(0x7fcfac061740 msgr2=0x7fcfac0b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 shutdown_connections 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 -- 172.21.15.97:0/490142146 wait complete. 2023-12-07T07:55:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:55.972+0000 7fcfb3b05640 1 librados: shutdown 2023-12-07T07:55:55.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.086+0000 7f6367839640 10 monclient: get_monmap_and_config 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.086+0000 7f6367839640 10 monclient: build_initial_monmap 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.086+0000 7f6367839640 10 monclient: monmap: 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:56.087741+0000 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:56.087741+0000 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:56.089 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:56.090 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:56.090 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:56.090 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:56.090 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 1 Processor -- start 2023-12-07T07:55:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 1 -- start start 2023-12-07T07:55:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 10 monclient: init 2023-12-07T07:55:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-12-07T07:55:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f63600a8910 0x7f636013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 10 monclient(hunting): picked mon.noname-a con 0x7f63600a8910 addr v2:172.21.15.97:3300/0 2023-12-07T07:55:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f63600a9b70 0x7f636013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 10 monclient(hunting): picked mon.noname-b con 0x7f63600a9b70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:56.093 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.087+0000 7f6367839640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6360141f90 0x7f6360142400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 10 monclient(hunting): picked mon.noname-c con 0x7f6360141f90 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600a8cf0 con 0x7f63600a8910 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636013f6e0 con 0x7f6360141f90 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600aabd0 con 0x7f63600a9b70 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6367839640 10 monclient(hunting): authenticate will time out at 2678.466325s 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6364dd4640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f63600a9b70 0x7f636013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.088+0000 7f6365dd6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6360141f90 0x7f6360142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f63600a9b70 0x7f636013f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:38902/0 (socket says 172.21.15.97:38902) 2023-12-07T07:55:56.094 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 1 -- 172.21.15.97:0/1176511482 learned_addr learned my addr 172.21.15.97:0/1176511482 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 10 monclient(hunting): get_auth_request con 0x7f63600a9b70 auth_method 0 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6365dd6640 10 monclient(hunting): get_auth_request con 0x7f6360141f90 auth_method 0 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6365dd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6365dd6640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6365dd6640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6365dd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6365dd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6365dd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:56.095 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.089+0000 7f6364dd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 10 monclient(hunting): handle_auth_done global_id 20263 payload 274 2023-12-07T07:55:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 1 -- 172.21.15.97:0/1176511482 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f63600a9b70 msgr2=0x7f636013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 1 --2- 172.21.15.97:0/1176511482 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f63600a9b70 0x7f636013f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 1 -- 172.21.15.97:0/1176511482 >> v2:172.21.15.97:3300/0 conn(0x7f63600a8910 msgr2=0x7f636013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 1 --2- 172.21.15.97:0/1176511482 >> v2:172.21.15.97:3300/0 conn(0x7f63600a8910 0x7f636013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 10 monclient: _finish_hunting 0 2023-12-07T07:55:56.096 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 1 monclient: found mon.noname-c 2023-12-07T07:55:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2023-12-07T07:55:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.091+0000 7f6365dd6640 1 -- 172.21.15.97:0/1176511482 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600aaeb0 con 0x7f6360141f90 2023-12-07T07:55:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.093+0000 7f6365dd6640 1 --2- 172.21.15.97:0/1176511482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360141f90 0x7f6360142400 secure :-1 s=READY pgs=4878 cs=0 l=1 rev1=1 crypto rx=0x7f63600a92b0 tx=0x7f635802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da786b61c5c5a0ea server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 1 -- 172.21.15.97:0/1176511482 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635802cbd0 con 0x7f6360141f90 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: dump: 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.099 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: _finish_auth 0 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: _check_auth_tickets 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 1 -- 172.21.15.97:0/1176511482 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635802cd30 con 0x7f6360141f90 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6367839640 5 monclient: authenticate success, global_id 20263 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6367839640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:56.100 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6367839640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6367839640 4 set_mon_vals no callback set 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 1 -- 172.21.15.97:0/1176511482 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635802cea0 con 0x7f6360141f90 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.096+0000 7f6357fff640 10 monclient: dump: 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.101 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.102 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 10 monclient: shutdown 2023-12-07T07:55:56.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:56.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 -- 172.21.15.97:0/1176511482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360141f90 msgr2=0x7f6360142400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 --2- 172.21.15.97:0/1176511482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360141f90 0x7f6360142400 secure :-1 s=READY pgs=4878 cs=0 l=1 rev1=1 crypto rx=0x7f63600a92b0 tx=0x7f635802be20 comp rx=0 tx=0).stop 2023-12-07T07:55:56.104 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 -- 172.21.15.97:0/1176511482 shutdown_connections 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 --2- 172.21.15.97:0/1176511482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360141f90 0x7f6360142400 unknown :-1 s=CLOSED pgs=4878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 --2- 172.21.15.97:0/1176511482 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f63600a9b70 0x7f636013f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 --2- 172.21.15.97:0/1176511482 >> v2:172.21.15.97:3300/0 conn(0x7f63600a8910 0x7f636013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 -- 172.21.15.97:0/1176511482 >> 172.21.15.97:0/1176511482 conn(0x7f63600b7780 msgr2=0x7f63600b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 -- 172.21.15.97:0/1176511482 shutdown_connections 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.097+0000 7f6367839640 1 -- 172.21.15.97:0/1176511482 wait complete. 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 10 monclient: build_initial_monmap 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 10 monclient: monmap: 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:56.105 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 1 librados: starting msgr at 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 1 librados: starting objecter 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 1 Processor -- start 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 1 -- start start 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 1 librados: setting wanted keys 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 1 librados: calling monclient init 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.098+0000 7f6367839640 10 monclient: init 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:56.106 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63600a8910 0x7f636013afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): picked mon.noname-a con 0x7f63600a8910 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6360141f90 0x7f636011fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): picked mon.noname-c con 0x7f6360141f90 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360120200 0x7f63601d7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): picked mon.noname-b con 0x7f6360120200 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636013ac70 con 0x7f63600a8910 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636013ade0 con 0x7f6360120200 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636011f6d0 con 0x7f6360141f90 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6367839640 10 monclient(hunting): authenticate will time out at 2678.477673s 2023-12-07T07:55:56.107 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6365dd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360120200 0x7f63601d7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.099+0000 7f6364dd4640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6360141f90 0x7f636011fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360120200 0x7f63601d7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.97:42212/0 (socket says 172.21.15.97:42212) 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 1 -- 172.21.15.97:0/3580200372 learned_addr learned my addr 172.21.15.97:0/3580200372 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 10 monclient(hunting): get_auth_request con 0x7f6360120200 auth_method 0 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6364dd4640 10 monclient(hunting): get_auth_request con 0x7f6360141f90 auth_method 0 2023-12-07T07:55:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6364dd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6364dd4640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6364dd4640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6365dd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6364dd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6364dd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:56.109 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.100+0000 7f6364dd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 10 monclient(hunting): handle_auth_done global_id 20266 payload 824 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 1 -- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6360141f90 msgr2=0x7f636011fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6360141f90 0x7f636011fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 1 -- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63600a8910 msgr2=0x7f636013afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63600a8910 0x7f636013afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 10 monclient: _finish_hunting 0 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 1 monclient: found mon.noname-b 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.103+0000 7f6365dd6640 1 -- 172.21.15.97:0/3580200372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358009a70 con 0x7f6360120200 2023-12-07T07:55:56.111 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.105+0000 7f6365dd6640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360120200 0x7f63601d7430 secure :-1 s=READY pgs=4879 cs=0 l=1 rev1=1 crypto rx=0x7f635802c430 tx=0x7f6358004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.105+0000 7f6355ffb640 10 client.?.objecter ms_handle_connect 0x7f6360120200 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.105+0000 7f6355ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6355ffb640 1 -- 172.21.15.97:0/3580200372 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635802cea0 con 0x7f6360120200 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6355ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6355ffb640 10 monclient: dump: 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.112 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6355ffb640 10 monclient: _finish_auth 0 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6355ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6355ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 5 monclient: authenticate success, global_id 20266 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 10 monclient: _renew_subs 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63601d7970 con 0x7f6360120200 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 10 client.20266.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 10 monclient: _renew_subs 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6360139fd0 con 0x7f6360120200 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.107+0000 7f6367839640 1 librados: init done 2023-12-07T07:55:56.113 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6367839640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6355ffb640 1 -- 172.21.15.97:0/3580200372 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635803e610 con 0x7f6360120200 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6367839640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6367839640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636011f3f0 con 0x7f6360120200 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6355ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6355ffb640 1 -- 172.21.15.97:0/3580200372 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635803e980 con 0x7f6360120200 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6355ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f6355ffb640 10 monclient: dump: 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.114 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.115 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.115 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.108+0000 7f63577fe640 4 set_mon_vals no callback set 2023-12-07T07:55:56.115 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:22.560 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.117 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:22.566 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.131 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:22.579 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.164 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:22.613 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.166 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:22.615 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.177 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:22.626 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.349+0000 7f6355ffb640 1 -- 172.21.15.97:0/3580200372 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f635803b050 con 0x7f6360120200 2023-12-07T07:55:56.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.349+0000 7f6355ffb640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f634406c7d0 0x7f634406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.350+0000 7f63655d5640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f634406c7d0 0x7f634406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.352 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.350+0000 7f63655d5640 10 monclient: get_auth_request con 0x7f634406c7d0 auth_method 0 2023-12-07T07:55:56.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.350+0000 7f6355ffb640 10 client.20266.objecter ms_handle_connect 0x7f634406c7d0 2023-12-07T07:55:56.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.350+0000 7f63655d5640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f634406c7d0 0x7f634406ec90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f635c009710 tx=0x7f635c006c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:56.398 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.396+0000 7f6355ffb640 1 -- 172.21.15.97:0/3580200372 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f635807afb0 con 0x7f6360120200 2023-12-07T07:55:56.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.396+0000 7f6355ffb640 10 client.20266.objecter ms_dispatch 0x7f6360000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:56.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.396+0000 7f6355ffb640 3 client.20266.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:56.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.396+0000 7f6355ffb640 3 client.20266.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:56.399 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.396+0000 7f6355ffb640 20 client.20266.objecter dump_active .. 0 homeless 2023-12-07T07:55:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.405+0000 7f6355ffb640 1 -- 172.21.15.97:0/3580200372 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63580772f0 con 0x7f6360120200 2023-12-07T07:55:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.405+0000 7f6355ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:56.408 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.405+0000 7f6355ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:56.411 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:22.849 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.415 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:22.853 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.426 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:22.863 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:55:56.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.590+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f63600a8fa0 con 0x7f634406c7d0 2023-12-07T07:55:56.593 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.591+0000 7f6355ffb640 1 -- 172.21.15.97:0/3580200372 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f635c00b480 con 0x7f634406c7d0 2023-12-07T07:55:56.593 INFO:teuthology.orchestra.run.smithi097.stderr:instructed osd(s) 7 to deep-scrub 2023-12-07T07:55:56.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.592+0000 7f6367839640 10 librados: watch_flush enter 2023-12-07T07:55:56.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.592+0000 7f6367839640 10 librados: watch_flush exit 2023-12-07T07:55:56.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.592+0000 7f6367839640 20 client.20266.objecter shutdown clearing up homeless session... 2023-12-07T07:55:56.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.592+0000 7f6367839640 10 client.20266.objecter successfully canceled tick 2023-12-07T07:55:56.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f634406c7d0 msgr2=0x7f634406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.596 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f634406c7d0 0x7f634406ec90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f635c009710 tx=0x7f635c006c90 comp rx=0 tx=0).stop 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 10 monclient: shutdown 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360120200 msgr2=0x7f63601d7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360120200 0x7f63601d7430 secure :-1 s=READY pgs=4879 cs=0 l=1 rev1=1 crypto rx=0x7f635802c430 tx=0x7f6358004a40 comp rx=0 tx=0).stop 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 shutdown_connections 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f634406c7d0 0x7f634406ec90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360120200 0x7f63601d7430 unknown :-1 s=CLOSED pgs=4879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6360141f90 0x7f636011fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 --2- 172.21.15.97:0/3580200372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f63600a8910 0x7f636013afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 >> 172.21.15.97:0/3580200372 conn(0x7f63600b7780 msgr2=0x7f63600b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 shutdown_connections 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 -- 172.21.15.97:0/3580200372 wait complete. 2023-12-07T07:55:56.597 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.593+0000 7f6367839640 1 librados: shutdown 2023-12-07T07:55:56.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:55:56.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.707+0000 7f4ff8075640 10 monclient: get_monmap_and_config 2023-12-07T07:55:56.709 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.707+0000 7f4ff8075640 10 monclient: build_initial_monmap 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.707+0000 7f4ff8075640 10 monclient: monmap: 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:55:56.708223+0000 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:55:56.708223+0000 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:55:56.710 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.711 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.707+0000 7f4ff8075640 1 Processor -- start 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.707+0000 7f4ff8075640 1 -- start start 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient: init 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ff00a8910 addr v1:172.21.15.154:6789/0 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4ff00a9b70 0x7f4ff013f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff00a9b70 addr v2:172.21.15.154:3300/0 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ff0141f90 addr v1:172.21.15.97:6789/0 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00a8cf0 con 0x7f4ff0141f90 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00aabd0 con 0x7f4ff00a8910 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff00aaeb0 con 0x7f4ff00a9b70 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:56.712 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.708+0000 7f4ff8075640 10 monclient(hunting): authenticate will time out at 2679.086923s 2023-12-07T07:55:56.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.713+0000 7f4ff6612640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4ff0141f90 0x7f4ff01423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47122/0 (socket says 172.21.15.97:47122) 2023-12-07T07:55:56.715 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.713+0000 7f4ff6612640 1 -- 172.21.15.97:0/3208925959 learned_addr learned my addr 172.21.15.97:0/3208925959 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:55:56.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.719+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613809088 0 0) 0x7f4ff00a8cf0 con 0x7f4ff0141f90 2023-12-07T07:55:56.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.719+0000 7f4ff4e0f640 10 monclient(hunting): _negotiate 2023-12-07T07:55:56.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.719+0000 7f4ff4e0f640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.719+0000 7f4ff4e0f640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:56.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.719+0000 7f4ff4e0f640 10 monclient(hunting): my global_id is 16215 2023-12-07T07:55:56.722 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.719+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003520 con 0x7f4ff0141f90 2023-12-07T07:55:56.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.728+0000 7f4ff5610640 1 --2- 172.21.15.97:0/3208925959 >> v2:172.21.15.154:3300/0 conn(0x7f4ff00a9b70 0x7f4ff013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165870938 0 0) 0x7f4fdc003520 con 0x7f4ff0141f90 2023-12-07T07:55:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 >> v2:172.21.15.154:3300/0 conn(0x7f4ff00a9b70 msgr2=0x7f4ff013f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 --2- 172.21.15.97:0/3208925959 >> v2:172.21.15.154:3300/0 conn(0x7f4ff00a9b70 0x7f4ff013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 >> v1:172.21.15.154:6789/0 conn(0x7f4ff00a8910 legacy=0x7f4ff013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:56.740 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: _finish_hunting 0 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 monclient: found mon.noname-a-legacy 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.97:6789/0 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00a9720 con 0x7f4ff0141f90 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: _finish_auth 0 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: _check_auth_tickets 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2176119509 0 0) 0x7f4fe80019f0 con 0x7f4ff0141f90 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff8075640 5 monclient: authenticate success, global_id 16215 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff8075640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-07T07:55:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: dump: 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] but i'm connected to v1:172.21.15.97:6789/0, reconnecting 2023-12-07T07:55:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 >> v1:172.21.15.97:6789/0 conn(0x7f4ff0141f90 legacy=0x7f4ff01423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fdc0039c0 0x7f4fdc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): picked mon.b con 0x7f4fdc0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0047c0 0x7f4fdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): picked mon.a con 0x7f4fdc0047c0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): picked mon.c con 0x7f4fdc0052a0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00aaeb0 con 0x7f4fdc0047c0 2023-12-07T07:55:56.743 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00c1f0 con 0x7f4fdc0039c0 2023-12-07T07:55:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00c3b0 con 0x7f4fdc0052a0 2023-12-07T07:55:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff4e0f640 10 monclient(hunting): _finish_auth 0 2023-12-07T07:55:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.737+0000 7f4ff8075640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.739+0000 7f4ff6612640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.741+0000 7f4ff6612640 10 monclient(hunting): get_auth_request con 0x7f4fdc0052a0 auth_method 0 2023-12-07T07:55:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.741+0000 7f4ff6612640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.741+0000 7f4ff6612640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.741+0000 7f4ff6612640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:55:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.743+0000 7f4ff6612640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.743+0000 7f4ff6612640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.743+0000 7f4ff6612640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:55:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.743+0000 7f4ff5610640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0047c0 0x7f4fdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.745+0000 7f4ff5610640 10 monclient(hunting): get_auth_request con 0x7f4fdc0047c0 auth_method 0 2023-12-07T07:55:56.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.745+0000 7f4ff5610640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:56.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.745+0000 7f4ff5610640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.747 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.745+0000 7f4ff5610640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-07T07:55:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff5610640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff5610640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff5610640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-07T07:55:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 10 monclient(hunting): handle_auth_done global_id 16215 payload 293 2023-12-07T07:55:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 1 -- 172.21.15.97:0/3208925959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fdc0039c0 msgr2=0x7f4fdc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fdc0039c0 0x7f4fdc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 1 -- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0047c0 msgr2=0x7f4fdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0047c0 0x7f4fdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 10 monclient: _finish_hunting 0 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 1 monclient: found mon.c 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 1 -- 172.21.15.97:0/3208925959 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc003520 con 0x7f4fdc0052a0 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 10 monclient: _finish_auth 0 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 10 monclient: _check_auth_tickets 2023-12-07T07:55:56.750 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.746+0000 7f4ff6612640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.747+0000 7f4ff6612640 1 -- 172.21.15.97:0/3208925959 reap_dead start 2023-12-07T07:55:56.753 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.751+0000 7f4ff6612640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800a7e0 tx=0x7f4fdc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d5712a3c9b611a0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:56.772 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.769+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe80072d0 con 0x7f4fdc0052a0 2023-12-07T07:55:56.772 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.772 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:56.772 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 10 monclient: dump: 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe802f070 con 0x7f4fdc0052a0 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 10 monclient: get_monmap_and_config success 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 4 set_mon_vals no callback set 2023-12-07T07:55:56.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 1 -- 172.21.15.97:0/3208925959 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8007670 con 0x7f4fdc0052a0 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff4e0f640 10 monclient: dump: 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.774 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 10 monclient: shutdown 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 1 -- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4fdc0052a0 msgr2=0x7f4fdc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800a7e0 tx=0x7f4fdc0057f0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 1 -- 172.21.15.97:0/3208925959 shutdown_connections 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 1 --2- 172.21.15.97:0/3208925959 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 1 -- 172.21.15.97:0/3208925959 >> 172.21.15.97:0/3208925959 conn(0x7f4ff00b7780 msgr2=0x7f4ff00b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 1 -- 172.21.15.97:0/3208925959 shutdown_connections 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.770+0000 7f4ff8075640 1 -- 172.21.15.97:0/3208925959 wait complete. 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.771+0000 7f4ff8075640 10 monclient: build_initial_monmap 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.771+0000 7f4ff8075640 10 monclient: monmap: 2023-12-07T07:55:56.777 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.771+0000 7f4ff8075640 1 librados: starting msgr at 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.771+0000 7f4ff8075640 1 librados: starting objecter 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 Processor -- start 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 -- start start 2023-12-07T07:55:56.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 librados: setting wanted keys 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 librados: calling monclient init 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient: init 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient: _reopen_session rank -1 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0052a0 0x7f4ff00b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): picked mon.noname-a con 0x7f4fdc0052a0 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff00b0c10 0x7f4ff01393a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff00b0c10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ff01398e0 0x7f4ff01d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff01398e0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff013a0b0 con 0x7f4fdc0052a0 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00aed50 con 0x7f4ff00b0c10 2023-12-07T07:55:56.779 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): start opening mon connection 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00af920 con 0x7f4ff01398e0 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): _renew_subs 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.772+0000 7f4ff8075640 10 monclient(hunting): authenticate will time out at 2679.150958s 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0052a0 0x7f4ff00b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0052a0 0x7f4ff00b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46038/0 (socket says 172.21.15.97:46038) 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 1 -- 172.21.15.97:0/129956765 learned_addr learned my addr 172.21.15.97:0/129956765 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 10 monclient(hunting): get_auth_request con 0x7f4fdc0052a0 auth_method 0 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.773+0000 7f4ff5e11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:55:56.780 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.774+0000 7f4ff5610640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff00b0c10 0x7f4ff01393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff6612640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ff01398e0 0x7f4ff01d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:56.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 10 monclient(hunting): handle_auth_done global_id 16218 payload 824 2023-12-07T07:55:56.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 1 -- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ff01398e0 msgr2=0x7f4ff01d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ff01398e0 0x7f4ff01d6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 1 -- 172.21.15.97:0/129956765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff00b0c10 msgr2=0x7f4ff01393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:56.782 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff00b0c10 0x7f4ff01393a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 10 monclient: _finish_hunting 0 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 1 monclient: found mon.noname-a 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.776+0000 7f4ff5e11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.777+0000 7f4ff5e11640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.777+0000 7f4ff5e11640 1 -- 172.21.15.97:0/129956765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff0139d90 con 0x7f4fdc0052a0 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.779+0000 7f4ff5e11640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0052a0 0x7f4ff00b06d0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c410 tx=0x7f4fec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.779+0000 7f4fe67fc640 10 client.?.objecter ms_handle_connect 0x7f4fdc0052a0 2023-12-07T07:55:56.783 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.779+0000 7f4fe67fc640 10 client.?.objecter resend_mon_ops 2023-12-07T07:55:56.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 1 -- 172.21.15.97:0/129956765 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0035f0 con 0x7f4fdc0052a0 2023-12-07T07:55:56.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:55:56.790 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: dump: 2023-12-07T07:55:56.790 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: _finish_auth 0 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: _check_auth_tickets 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 1 -- 172.21.15.97:0/129956765 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec003750 con 0x7f4fdc0052a0 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 5 monclient: authenticate success, global_id 16218 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 10 monclient: _renew_subs 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:56.792 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff01d7350 con 0x7f4fdc0052a0 2023-12-07T07:55:56.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:55:56.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 1 -- 172.21.15.97:0/129956765 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec003b40 con 0x7f4fdc0052a0 2023-12-07T07:55:56.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:55:56.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:55:56.794 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe67fc640 10 monclient: dump: 2023-12-07T07:55:56.794 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:55:56.795 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:55:56.795 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.795 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:55:56.795 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:55:56.795 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:55:56.795 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:55:56.795 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:55:56.796 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:55:56.796 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:55:56.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4fe7fff640 4 set_mon_vals no callback set 2023-12-07T07:55:56.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 10 client.16218.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:55:56.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 10 monclient: _renew_subs 2023-12-07T07:55:56.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:56.796 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff01d7810 con 0x7f4fdc0052a0 2023-12-07T07:55:56.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.785+0000 7f4ff8075640 1 librados: init done 2023-12-07T07:55:56.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.786+0000 7f4ff8075640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:56.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.786+0000 7f4ff8075640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:56.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.786+0000 7f4ff8075640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:55:56.797 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:56.786+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff013a220 con 0x7f4fdc0052a0 2023-12-07T07:55:57.035 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.033+0000 7f4fe67fc640 1 -- 172.21.15.97:0/129956765 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4fec003ca0 con 0x7f4fdc0052a0 2023-12-07T07:55:57.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.033+0000 7f4fe67fc640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4fd406c850 0x7f4fd406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:55:57.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.033+0000 7f4ff5610640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4fd406c850 0x7f4fd406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:55:57.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.033+0000 7f4ff5610640 10 monclient: get_auth_request con 0x7f4fd406c850 auth_method 0 2023-12-07T07:55:57.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.033+0000 7f4fe67fc640 10 client.16218.objecter ms_handle_connect 0x7f4fd406c850 2023-12-07T07:55:57.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.033+0000 7f4ff5610640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4fd406c850 0x7f4fd406ed10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00ae9d0 tx=0x7f4fe0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:55:57.047 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 1 -- 172.21.15.97:0/129956765 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f4fec058450 con 0x7f4fdc0052a0 2023-12-07T07:55:57.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 10 client.16218.objecter ms_dispatch 0x7f4ff0000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:55:57.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 3 client.16218.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:55:57.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 3 client.16218.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:55:57.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 20 client.16218.objecter dump_active .. 0 homeless 2023-12-07T07:55:57.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 1 -- 172.21.15.97:0/129956765 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fec054090 con 0x7f4fdc0052a0 2023-12-07T07:55:57.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:55:57.048 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.045+0000 7f4fe67fc640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:55:57.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.217+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ff01d7a30 con 0x7f4fd406c850 2023-12-07T07:55:57.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.222+0000 7f4fe67fc640 1 -- 172.21.15.97:0/129956765 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151375 (secure 0 0 0) 0x7f4ff01d7a30 con 0x7f4fd406c850 2023-12-07T07:55:57.225 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:55:57.226 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 10 librados: watch_flush enter 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 10 librados: watch_flush exit 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 20 client.16218.objecter shutdown clearing up homeless session... 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 10 client.16218.objecter successfully canceled tick 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4fd406c850 msgr2=0x7f4fd406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4fd406c850 0x7f4fd406ed10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00ae9d0 tx=0x7f4fe0008040 comp rx=0 tx=0).stop 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 10 monclient: shutdown 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0052a0 msgr2=0x7f4ff00b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0052a0 0x7f4ff00b06d0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c410 tx=0x7f4fec00c8d0 comp rx=0 tx=0).stop 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 shutdown_connections 2023-12-07T07:55:57.228 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f4fd406c850 0x7f4fd406ed10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:57.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ff01398e0 0x7f4ff01d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:57.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff00b0c10 0x7f4ff01393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:57.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 --2- 172.21.15.97:0/129956765 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4fdc0052a0 0x7f4ff00b06d0 unknown :-1 s=CLOSED pgs=3240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:55:57.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 >> 172.21.15.97:0/129956765 conn(0x7f4ff00b7780 msgr2=0x7f4ff0142a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:55:57.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 shutdown_connections 2023-12-07T07:55:57.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 -- 172.21.15.97:0/129956765 wait complete. 2023-12-07T07:55:57.229 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:55:57.224+0000 7f4ff8075640 1 librados: shutdown 2023-12-07T07:55:57.237 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":1595,"stamp":"2023-12-07T07:55:55.964964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542984,"kb_used_data":53400,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754431736,"statfs":{"total":773094113280,"available":772538097664,"internally_reserved":0,"allocated":54681600,"data_stored":30217154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":501206576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001883"},"pg_stats":[{"pgid":"11.17","version":"0'0","reported_seq":865,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.122242+0000","last_change":"2023-12-07T07:34:49.234367+0000","last_active":"2023-12-07T07:55:46.122242+0000","last_peered":"2023-12-07T07:55:46.122242+0000","last_clean":"2023-12-07T07:55:46.122242+0000","last_became_active":"2023-12-07T07:34:49.221580+0000","last_became_peered":"2023-12-07T07:34:49.221580+0000","last_unstale":"2023-12-07T07:55:46.122242+0000","last_undegraded":"2023-12-07T07:55:46.122242+0000","last_fullsized":"2023-12-07T07:55:46.122242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":865,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.330964+0000","last_change":"2023-12-07T07:34:47.569420+0000","last_active":"2023-12-07T07:55:44.330964+0000","last_peered":"2023-12-07T07:55:44.330964+0000","last_clean":"2023-12-07T07:55:44.330964+0000","last_became_active":"2023-12-07T07:34:47.563950+0000","last_became_peered":"2023-12-07T07:34:47.563950+0000","last_unstale":"2023-12-07T07:55:44.330964+0000","last_undegraded":"2023-12-07T07:55:44.330964+0000","last_fullsized":"2023-12-07T07:55:44.330964+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:43.199927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":878,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.524025+0000","last_change":"2023-12-07T07:34:47.040719+0000","last_active":"2023-12-07T07:55:46.524025+0000","last_peered":"2023-12-07T07:55:46.524025+0000","last_clean":"2023-12-07T07:55:46.524025+0000","last_became_active":"2023-12-07T07:34:47.035305+0000","last_became_peered":"2023-12-07T07:34:47.035305+0000","last_unstale":"2023-12-07T07:55:46.524025+0000","last_undegraded":"2023-12-07T07:55:46.524025+0000","last_fullsized":"2023-12-07T07:55:46.524025+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:31:49.037782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":883,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:49.874362+0000","last_change":"2023-12-07T07:34:55.054864+0000","last_active":"2023-12-07T07:55:49.874362+0000","last_peered":"2023-12-07T07:55:49.874362+0000","last_clean":"2023-12-07T07:55:49.874362+0000","last_became_active":"2023-12-07T07:34:55.049584+0000","last_became_peered":"2023-12-07T07:34:55.049584+0000","last_unstale":"2023-12-07T07:55:49.874362+0000","last_undegraded":"2023-12-07T07:55:49.874362+0000","last_fullsized":"2023-12-07T07:55:49.874362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":866,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.382695+0000","last_change":"2023-12-07T07:35:04.485603+0000","last_active":"2023-12-07T07:55:46.382695+0000","last_peered":"2023-12-07T07:55:46.382695+0000","last_clean":"2023-12-07T07:55:46.382695+0000","last_became_active":"2023-12-07T07:35:04.480195+0000","last_became_peered":"2023-12-07T07:35:04.480195+0000","last_unstale":"2023-12-07T07:55:46.382695+0000","last_undegraded":"2023-12-07T07:55:46.382695+0000","last_fullsized":"2023-12-07T07:55:46.382695+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:46:21.908088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":878,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:54.451518+0000","last_change":"2023-12-07T07:55:54.452220+0000","last_active":"2023-12-07T07:55:54.451518+0000","last_peered":"2023-12-07T07:55:54.451518+0000","last_clean":"2023-12-07T07:55:54.451518+0000","last_became_active":"2023-12-07T07:34:51.735923+0000","last_became_peered":"2023-12-07T07:34:51.735923+0000","last_unstale":"2023-12-07T07:55:54.451518+0000","last_undegraded":"2023-12-07T07:55:54.451518+0000","last_fullsized":"2023-12-07T07:55:54.451518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:54.447971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:54.447971+0000","last_clean_scrub_stamp":"2023-12-07T07:55:54.447971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:00:50.135085+0000","scrub_duration":0.041230276000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":891,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.062176+0000","last_change":"2023-12-07T07:34:47.067218+0000","last_active":"2023-12-07T07:55:46.062176+0000","last_peered":"2023-12-07T07:55:46.062176+0000","last_clean":"2023-12-07T07:55:46.062176+0000","last_became_active":"2023-12-07T07:34:47.052848+0000","last_became_peered":"2023-12-07T07:34:47.052848+0000","last_unstale":"2023-12-07T07:55:46.062176+0000","last_undegraded":"2023-12-07T07:55:46.062176+0000","last_fullsized":"2023-12-07T07:55:46.062176+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:34:00.206441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":879,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.569736+0000","last_change":"2023-12-07T07:34:51.124246+0000","last_active":"2023-12-07T07:55:46.569736+0000","last_peered":"2023-12-07T07:55:46.569736+0000","last_clean":"2023-12-07T07:55:46.569736+0000","last_became_active":"2023-12-07T07:34:51.115413+0000","last_became_peered":"2023-12-07T07:34:51.115413+0000","last_unstale":"2023-12-07T07:55:46.569736+0000","last_undegraded":"2023-12-07T07:55:46.569736+0000","last_fullsized":"2023-12-07T07:55:46.569736+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:15:19.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":892,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.963691+0000","last_change":"2023-12-07T07:34:47.170254+0000","last_active":"2023-12-07T07:55:45.963691+0000","last_peered":"2023-12-07T07:55:45.963691+0000","last_clean":"2023-12-07T07:55:45.963691+0000","last_became_active":"2023-12-07T07:34:47.149617+0000","last_became_peered":"2023-12-07T07:34:47.149617+0000","last_unstale":"2023-12-07T07:55:45.963691+0000","last_undegraded":"2023-12-07T07:55:45.963691+0000","last_fullsized":"2023-12-07T07:55:45.963691+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:02:51.144651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":884,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.582790+0000","last_change":"2023-12-07T07:34:53.278594+0000","last_active":"2023-12-07T07:55:46.582790+0000","last_peered":"2023-12-07T07:55:46.582790+0000","last_clean":"2023-12-07T07:55:46.582790+0000","last_became_active":"2023-12-07T07:34:53.269724+0000","last_became_peered":"2023-12-07T07:34:53.269724+0000","last_unstale":"2023-12-07T07:55:46.582790+0000","last_undegraded":"2023-12-07T07:55:46.582790+0000","last_fullsized":"2023-12-07T07:55:46.582790+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":865,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.315267+0000","last_change":"2023-12-07T07:34:56.759465+0000","last_active":"2023-12-07T07:55:44.315267+0000","last_peered":"2023-12-07T07:55:44.315267+0000","last_clean":"2023-12-07T07:55:44.315267+0000","last_became_active":"2023-12-07T07:34:56.746142+0000","last_became_peered":"2023-12-07T07:34:56.746142+0000","last_unstale":"2023-12-07T07:55:44.315267+0000","last_undegraded":"2023-12-07T07:55:44.315267+0000","last_fullsized":"2023-12-07T07:55:44.315267+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T16:37:43.750416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":869,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.535815+0000","last_change":"2023-12-07T07:34:49.363608+0000","last_active":"2023-12-07T07:55:46.535815+0000","last_peered":"2023-12-07T07:55:46.535815+0000","last_clean":"2023-12-07T07:55:46.535815+0000","last_became_active":"2023-12-07T07:34:49.356256+0000","last_became_peered":"2023-12-07T07:34:49.356256+0000","last_unstale":"2023-12-07T07:55:46.535815+0000","last_undegraded":"2023-12-07T07:55:46.535815+0000","last_fullsized":"2023-12-07T07:55:46.535815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.593684+0000","last_change":"2023-12-07T07:33:42.670811+0000","last_active":"2023-12-07T07:55:46.593684+0000","last_peered":"2023-12-07T07:55:46.593684+0000","last_clean":"2023-12-07T07:55:46.593684+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:55:46.593684+0000","last_undegraded":"2023-12-07T07:55:46.593684+0000","last_fullsized":"2023-12-07T07:55:46.593684+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":866,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.094505+0000","last_change":"2023-12-07T07:34:48.180457+0000","last_active":"2023-12-07T07:55:46.094505+0000","last_peered":"2023-12-07T07:55:46.094505+0000","last_clean":"2023-12-07T07:55:46.094505+0000","last_became_active":"2023-12-07T07:34:48.171478+0000","last_became_peered":"2023-12-07T07:34:48.171478+0000","last_unstale":"2023-12-07T07:55:46.094505+0000","last_undegraded":"2023-12-07T07:55:46.094505+0000","last_fullsized":"2023-12-07T07:55:46.094505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":878,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.913529+0000","last_change":"2023-12-07T07:33:43.216945+0000","last_active":"2023-12-07T07:55:45.913529+0000","last_peered":"2023-12-07T07:55:45.913529+0000","last_clean":"2023-12-07T07:55:45.913529+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:55:45.913529+0000","last_undegraded":"2023-12-07T07:55:45.913529+0000","last_fullsized":"2023-12-07T07:55:45.913529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:54.221713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":866,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.117889+0000","last_change":"2023-12-07T07:34:49.998279+0000","last_active":"2023-12-07T07:55:46.117889+0000","last_peered":"2023-12-07T07:55:46.117889+0000","last_clean":"2023-12-07T07:55:46.117889+0000","last_became_active":"2023-12-07T07:34:49.980936+0000","last_became_peered":"2023-12-07T07:34:49.980936+0000","last_unstale":"2023-12-07T07:55:46.117889+0000","last_undegraded":"2023-12-07T07:55:46.117889+0000","last_fullsized":"2023-12-07T07:55:46.117889+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.585491+0000","last_change":"2023-12-07T07:33:42.426643+0000","last_active":"2023-12-07T07:55:46.585491+0000","last_peered":"2023-12-07T07:55:46.585491+0000","last_clean":"2023-12-07T07:55:46.585491+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:55:46.585491+0000","last_undegraded":"2023-12-07T07:55:46.585491+0000","last_fullsized":"2023-12-07T07:55:46.585491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":891,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.965586+0000","last_change":"2023-12-07T07:34:51.761228+0000","last_active":"2023-12-07T07:55:45.965586+0000","last_peered":"2023-12-07T07:55:45.965586+0000","last_clean":"2023-12-07T07:55:45.965586+0000","last_became_active":"2023-12-07T07:34:51.754214+0000","last_became_peered":"2023-12-07T07:34:51.754214+0000","last_unstale":"2023-12-07T07:55:45.965586+0000","last_undegraded":"2023-12-07T07:55:45.965586+0000","last_fullsized":"2023-12-07T07:55:45.965586+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:05:43.242940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":884,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.522315+0000","last_change":"2023-12-07T07:33:42.656077+0000","last_active":"2023-12-07T07:55:46.522315+0000","last_peered":"2023-12-07T07:55:46.522315+0000","last_clean":"2023-12-07T07:55:46.522315+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:55:46.522315+0000","last_undegraded":"2023-12-07T07:55:46.522315+0000","last_fullsized":"2023-12-07T07:55:46.522315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:05:05.686340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":891,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.961806+0000","last_change":"2023-12-07T07:35:04.576294+0000","last_active":"2023-12-07T07:55:45.961806+0000","last_peered":"2023-12-07T07:55:45.961806+0000","last_clean":"2023-12-07T07:55:45.961806+0000","last_became_active":"2023-12-07T07:35:04.566145+0000","last_became_peered":"2023-12-07T07:35:04.566145+0000","last_unstale":"2023-12-07T07:55:45.961806+0000","last_undegraded":"2023-12-07T07:55:45.961806+0000","last_fullsized":"2023-12-07T07:55:45.961806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:07:29.588414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.568182+0000","last_change":"2023-12-07T07:33:42.617026+0000","last_active":"2023-12-07T07:55:46.568182+0000","last_peered":"2023-12-07T07:55:46.568182+0000","last_clean":"2023-12-07T07:55:46.568182+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:55:46.568182+0000","last_undegraded":"2023-12-07T07:55:46.568182+0000","last_fullsized":"2023-12-07T07:55:46.568182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":872,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.372731+0000","last_change":"2023-12-07T07:34:47.445712+0000","last_active":"2023-12-07T07:55:46.372731+0000","last_peered":"2023-12-07T07:55:46.372731+0000","last_clean":"2023-12-07T07:55:46.372731+0000","last_became_active":"2023-12-07T07:34:47.437980+0000","last_became_peered":"2023-12-07T07:34:47.437980+0000","last_unstale":"2023-12-07T07:55:46.372731+0000","last_undegraded":"2023-12-07T07:55:46.372731+0000","last_fullsized":"2023-12-07T07:55:46.372731+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:52:30.429862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":896,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.956192+0000","last_change":"2023-12-07T07:33:42.269533+0000","last_active":"2023-12-07T07:55:45.956192+0000","last_peered":"2023-12-07T07:55:45.956192+0000","last_clean":"2023-12-07T07:55:45.956192+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:55:45.956192+0000","last_undegraded":"2023-12-07T07:55:45.956192+0000","last_fullsized":"2023-12-07T07:55:45.956192+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:12:50.238470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":872,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.358163+0000","last_change":"2023-12-07T07:35:04.029478+0000","last_active":"2023-12-07T07:55:46.358163+0000","last_peered":"2023-12-07T07:55:46.358163+0000","last_clean":"2023-12-07T07:55:46.358163+0000","last_became_active":"2023-12-07T07:35:04.015545+0000","last_became_peered":"2023-12-07T07:35:04.015545+0000","last_unstale":"2023-12-07T07:55:46.358163+0000","last_undegraded":"2023-12-07T07:55:46.358163+0000","last_fullsized":"2023-12-07T07:55:46.358163+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:16:10.231382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":897,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:53.453695+0000","last_change":"2023-12-07T07:55:53.453695+0000","last_active":"2023-12-07T07:55:53.453695+0000","last_peered":"2023-12-07T07:55:53.453695+0000","last_clean":"2023-12-07T07:55:53.453695+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:55:53.453695+0000","last_undegraded":"2023-12-07T07:55:53.453695+0000","last_fullsized":"2023-12-07T07:55:53.453695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:53.451589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:53.451589+0000","last_clean_scrub_stamp":"2023-12-07T07:55:53.451589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:55:36.310408+0000","scrub_duration":0.050438442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":876,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.907529+0000","last_change":"2023-12-07T07:34:52.478105+0000","last_active":"2023-12-07T07:55:45.907529+0000","last_peered":"2023-12-07T07:55:45.907529+0000","last_clean":"2023-12-07T07:55:45.907529+0000","last_became_active":"2023-12-07T07:34:52.469358+0000","last_became_peered":"2023-12-07T07:34:52.469358+0000","last_unstale":"2023-12-07T07:55:45.907529+0000","last_undegraded":"2023-12-07T07:55:45.907529+0000","last_fullsized":"2023-12-07T07:55:45.907529+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:11:55.093716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.574020+0000","last_change":"2023-12-07T07:33:42.649680+0000","last_active":"2023-12-07T07:55:46.574020+0000","last_peered":"2023-12-07T07:55:46.574020+0000","last_clean":"2023-12-07T07:55:46.574020+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:55:46.574020+0000","last_undegraded":"2023-12-07T07:55:46.574020+0000","last_fullsized":"2023-12-07T07:55:46.574020+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:40.740374+0000","last_clean_scrub_stamp":"2023-12-07T07:33:40.740374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'76","reported_seq":978,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.370350+0000","last_change":"2023-12-07T07:33:36.384672+0000","last_active":"2023-12-07T07:55:46.370350+0000","last_peered":"2023-12-07T07:55:46.370350+0000","last_clean":"2023-12-07T07:55:46.370350+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:55:46.370350+0000","last_undegraded":"2023-12-07T07:55:46.370350+0000","last_fullsized":"2023-12-07T07:55:46.370350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:33:30.679434+0000","last_clean_scrub_stamp":"2023-12-07T07:33:30.679434+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:54:14.180629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.519129+0000","last_change":"2023-12-07T07:34:47.016526+0000","last_active":"2023-12-07T07:55:46.519129+0000","last_peered":"2023-12-07T07:55:46.519129+0000","last_clean":"2023-12-07T07:55:46.519129+0000","last_became_active":"2023-12-07T07:34:47.010508+0000","last_became_peered":"2023-12-07T07:34:47.010508+0000","last_unstale":"2023-12-07T07:55:46.519129+0000","last_undegraded":"2023-12-07T07:55:46.519129+0000","last_fullsized":"2023-12-07T07:55:46.519129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:26:28.206752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":873,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.360326+0000","last_change":"2023-12-07T07:35:02.508081+0000","last_active":"2023-12-07T07:55:46.360326+0000","last_peered":"2023-12-07T07:55:46.360326+0000","last_clean":"2023-12-07T07:55:46.360326+0000","last_became_active":"2023-12-07T07:35:02.498406+0000","last_became_peered":"2023-12-07T07:35:02.498406+0000","last_unstale":"2023-12-07T07:55:46.360326+0000","last_undegraded":"2023-12-07T07:55:46.360326+0000","last_fullsized":"2023-12-07T07:55:46.360326+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:26:16.927736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1","version":"38'7","reported_seq":881,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.086288+0000","last_change":"2023-12-07T07:34:48.573706+0000","last_active":"2023-12-07T07:55:46.086288+0000","last_peered":"2023-12-07T07:55:46.086288+0000","last_clean":"2023-12-07T07:55:46.086288+0000","last_became_active":"2023-12-07T07:34:48.562400+0000","last_became_peered":"2023-12-07T07:34:48.562400+0000","last_unstale":"2023-12-07T07:55:46.086288+0000","last_undegraded":"2023-12-07T07:55:46.086288+0000","last_fullsized":"2023-12-07T07:55:46.086288+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":880,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.520737+0000","last_change":"2023-12-07T07:34:50.852474+0000","last_active":"2023-12-07T07:55:46.520737+0000","last_peered":"2023-12-07T07:55:46.520737+0000","last_clean":"2023-12-07T07:55:46.520737+0000","last_became_active":"2023-12-07T07:34:50.845529+0000","last_became_peered":"2023-12-07T07:34:50.845529+0000","last_unstale":"2023-12-07T07:55:46.520737+0000","last_undegraded":"2023-12-07T07:55:46.520737+0000","last_fullsized":"2023-12-07T07:55:46.520737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:04:08.453958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":884,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.571121+0000","last_change":"2023-12-07T07:34:44.988549+0000","last_active":"2023-12-07T07:55:46.571121+0000","last_peered":"2023-12-07T07:55:46.571121+0000","last_clean":"2023-12-07T07:55:46.571121+0000","last_became_active":"2023-12-07T07:34:44.899627+0000","last_became_peered":"2023-12-07T07:34:44.899627+0000","last_unstale":"2023-12-07T07:55:46.571121+0000","last_undegraded":"2023-12-07T07:55:46.571121+0000","last_fullsized":"2023-12-07T07:55:46.571121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":868,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.091764+0000","last_change":"2023-12-07T07:34:50.497421+0000","last_active":"2023-12-07T07:55:46.091764+0000","last_peered":"2023-12-07T07:55:46.091764+0000","last_clean":"2023-12-07T07:55:46.091764+0000","last_became_active":"2023-12-07T07:34:50.418281+0000","last_became_peered":"2023-12-07T07:34:50.418281+0000","last_unstale":"2023-12-07T07:55:46.091764+0000","last_undegraded":"2023-12-07T07:55:46.091764+0000","last_fullsized":"2023-12-07T07:55:46.091764+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":867,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.316976+0000","last_change":"2023-12-07T07:34:58.289194+0000","last_active":"2023-12-07T07:55:44.316976+0000","last_peered":"2023-12-07T07:55:44.316976+0000","last_clean":"2023-12-07T07:55:44.316976+0000","last_became_active":"2023-12-07T07:34:58.263210+0000","last_became_peered":"2023-12-07T07:34:58.263210+0000","last_unstale":"2023-12-07T07:55:44.316976+0000","last_undegraded":"2023-12-07T07:55:44.316976+0000","last_fullsized":"2023-12-07T07:55:44.316976+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:45:56.554479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":879,"reported_epoch":918,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.529472+0000","last_change":"2023-12-07T07:34:47.231473+0000","last_active":"2023-12-07T07:55:46.529472+0000","last_peered":"2023-12-07T07:55:46.529472+0000","last_clean":"2023-12-07T07:55:46.529472+0000","last_became_active":"2023-12-07T07:34:47.225369+0000","last_became_peered":"2023-12-07T07:34:47.225369+0000","last_unstale":"2023-12-07T07:55:46.529472+0000","last_undegraded":"2023-12-07T07:55:46.529472+0000","last_fullsized":"2023-12-07T07:55:46.529472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:53:23.919797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":866,"reported_epoch":917,"state":"active+clean","last_fresh":"2023-12-07T07:55:44.313417+0000","last_change":"2023-12-07T07:35:00.370860+0000","last_active":"2023-12-07T07:55:44.313417+0000","last_peered":"2023-12-07T07:55:44.313417+0000","last_clean":"2023-12-07T07:55:44.313417+0000","last_became_active":"2023-12-07T07:35:00.363892+0000","last_became_peered":"2023-12-07T07:35:00.363892+0000","last_unstale":"2023-12-07T07:55:44.313417+0000","last_undegraded":"2023-12-07T07:55:44.313417+0000","last_fullsized":"2023-12-07T07:55:44.313417+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:52:59.781747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":871,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.355969+0000","last_change":"2023-12-07T07:34:45.545881+0000","last_active":"2023-12-07T07:55:46.355969+0000","last_peered":"2023-12-07T07:55:46.355969+0000","last_clean":"2023-12-07T07:55:46.355969+0000","last_became_active":"2023-12-07T07:34:45.533595+0000","last_became_peered":"2023-12-07T07:34:45.533595+0000","last_unstale":"2023-12-07T07:55:46.355969+0000","last_undegraded":"2023-12-07T07:55:46.355969+0000","last_fullsized":"2023-12-07T07:55:46.355969+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:06:30.943446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":875,"reported_epoch":919,"state":"active+clean","last_fresh":"2023-12-07T07:55:45.905930+0000","last_change":"2023-12-07T07:34:54.924870+0000","last_active":"2023-12-07T07:55:45.905930+0000","last_peered":"2023-12-07T07:55:45.905930+0000","last_clean":"2023-12-07T07:55:45.905930+0000","last_became_active":"2023-12-07T07:34:54.918404+0000","last_became_peered":"2023-12-07T07:34:54.918404+0000","last_unstale":"2023-12-07T07:55:45.905930+0000","last_undegraded":"2023-12-07T07:55:45.905930+0000","last_fullsized":"2023-12-07T07:55:45.905930+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:07:29.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":882,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.566500+0000","last_change":"2023-12-07T07:34:45.422416+0000","last_active":"2023-12-07T07:55:46.566500+0000","last_peered":"2023-12-07T07:55:46.566500+0000","last_clean":"2023-12-07T07:55:46.566500+0000","last_became_active":"2023-12-07T07:34:45.408105+0000","last_became_peered":"2023-12-07T07:34:45.408105+0000","last_unstale":"2023-12-07T07:55:46.566500+0000","last_undegraded":"2023-12-07T07:55:46.566500+0000","last_fullsized":"2023-12-07T07:55:46.566500+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":877,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:55.492980+0000","last_change":"2023-12-07T07:55:55.493639+0000","last_active":"2023-12-07T07:55:55.492980+0000","last_peered":"2023-12-07T07:55:55.492980+0000","last_clean":"2023-12-07T07:55:55.492980+0000","last_became_active":"2023-12-07T07:34:49.378907+0000","last_became_peered":"2023-12-07T07:34:49.378907+0000","last_unstale":"2023-12-07T07:55:55.492980+0000","last_undegraded":"2023-12-07T07:55:55.492980+0000","last_fullsized":"2023-12-07T07:55:55.492980+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:55.491658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:55.491658+0000","last_clean_scrub_stamp":"2023-12-07T07:55:55.491658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:28:12.211489+0000","scrub_duration":0.058683120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444301,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":25.919,"15min":28.937999999999999},"min":{"1min":6.7430000000000003,"5min":5.7389999999999999,"15min":3.8220000000000001},"max":{"1min":57.213999999999999,"5min":240.48099999999999,"15min":963.553},"last":10.945},{"interface":"front","average":{"1min":21.895,"5min":32.722000000000001,"15min":26.271999999999998},"min":{"1min":8.7810000000000006,"5min":8.7810000000000006,"15min":4.7640000000000002},"max":{"1min":123.477,"5min":252.37899999999999,"15min":285.09399999999999},"last":12.16}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":9.9960000000000004,"5min":20.056000000000001,"15min":19.806999999999999},"min":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":5.5049999999999999},"max":{"1min":35.368000000000002,"5min":236.608,"15min":240.595},"last":7.6420000000000003},{"interface":"front","average":{"1min":15.894,"5min":28.577000000000002,"15min":25.738},"min":{"1min":9.3689999999999998,"5min":9.0139999999999993,"15min":6.3490000000000002},"max":{"1min":33.170000000000002,"5min":275.33300000000003,"15min":307.64100000000002},"last":11.509}]},{"osd":2,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.742000000000001,"5min":29.213999999999999,"15min":23.890999999999998},"min":{"1min":10.449,"5min":9.9049999999999994,"15min":7.4749999999999996},"max":{"1min":56.265000000000001,"5min":277.43099999999998,"15min":277.43099999999998},"last":12.965999999999999},{"interface":"front","average":{"1min":12.913,"5min":22.611999999999998,"15min":24.943000000000001},"min":{"1min":6.3700000000000001,"5min":6.3700000000000001,"15min":6.3700000000000001},"max":{"1min":51.895000000000003,"5min":243.97800000000001,"15min":243.97800000000001},"last":10.343}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.201000000000001,"5min":21.512,"15min":22.870000000000001},"min":{"1min":5.7670000000000003,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":37.640999999999998,"5min":195.59299999999999,"15min":314.84300000000002},"last":6.8609999999999998},{"interface":"front","average":{"1min":18.228000000000002,"5min":27.143000000000001,"15min":26.853999999999999},"min":{"1min":9.6170000000000009,"5min":7.1500000000000004,"15min":7.1500000000000004},"max":{"1min":66.102999999999994,"5min":215.51900000000001,"15min":252.49799999999999},"last":16.995999999999999}]},{"osd":5,"last update":"Thu Dec 7 07:55:22 2023","interfaces":[{"interface":"back","average":{"1min":19.082999999999998,"5min":28.425999999999998,"15min":28.632999999999999},"min":{"1min":7.2640000000000002,"5min":7.2640000000000002,"15min":7.2640000000000002},"max":{"1min":76.856999999999999,"5min":241.82300000000001,"15min":241.82300000000001},"last":19.972000000000001},{"interface":"front","average":{"1min":20.713000000000001,"5min":30.940000000000001,"15min":30.433},"min":{"1min":9.3460000000000001,"5min":9.3460000000000001,"15min":8.6780000000000008},"max":{"1min":57.811,"5min":286.17200000000003,"15min":286.17200000000003},"last":13.555999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.530999999999999,"5min":21.175999999999998,"15min":26.318000000000001},"min":{"1min":7.0590000000000002,"5min":6.149,"15min":6.149},"max":{"1min":84.808000000000007,"5min":168.52600000000001,"15min":333.97000000000003},"last":8.7430000000000003},{"interface":"front","average":{"1min":19.992000000000001,"5min":28.367000000000001,"15min":28.050999999999998},"min":{"1min":10.449999999999999,"5min":10.449999999999999,"15min":9.0559999999999992},"max":{"1min":84.061999999999998,"5min":168.88499999999999,"15min":301.428},"last":18.056000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":24.667000000000002,"5min":28.010000000000002,"15min":28.553999999999998},"min":{"1min":12.593999999999999,"5min":8.9139999999999997,"15min":8.0549999999999997},"max":{"1min":83.186000000000007,"5min":186.81299999999999,"15min":247.31100000000001},"last":14.118},{"interface":"front","average":{"1min":19.212,"5min":20.692,"15min":24.864999999999998},"min":{"1min":8.0640000000000001,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":61.241999999999997,"5min":184.453,"15min":311.41000000000003},"last":9.3260000000000005}]}]},{"osd":2,"up_from":17,"seq":73014444301,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64120,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307720,"statfs":{"total":96636764160,"available":96571105280,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":17.835999999999999,"5min":15.121,"15min":16.925000000000001},"min":{"1min":5.9960000000000004,"5min":5.9960000000000004,"15min":4.9409999999999998},"max":{"1min":111.95099999999999,"5min":149.566,"15min":339.37299999999999},"last":8.8559999999999999},{"interface":"front","average":{"1min":21.946999999999999,"5min":20.587,"15min":23.597999999999999},"min":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":4.6319999999999997},"max":{"1min":123.776,"5min":157.69300000000001,"15min":597.92399999999998},"last":13.561999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.486000000000001,"5min":18.428000000000001,"15min":21.236999999999998},"min":{"1min":7.4930000000000003,"5min":7.4930000000000003,"15min":4.5410000000000004},"max":{"1min":58.052999999999997,"5min":104.455,"15min":282.435},"last":15.804},{"interface":"front","average":{"1min":13.083,"5min":13.429,"15min":18.023},"min":{"1min":5.093,"5min":5.093,"15min":5.093},"max":{"1min":51.125,"5min":100.026,"15min":135.71000000000001},"last":7.1630000000000003}]},{"osd":3,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":21.981999999999999,"5min":23.928000000000001,"15min":20.510999999999999},"min":{"1min":10.015000000000001,"5min":8.1869999999999994,"15min":6.1699999999999999},"max":{"1min":47.886000000000003,"5min":160.94800000000001,"15min":160.94800000000001},"last":27.934999999999999},{"interface":"front","average":{"1min":16.581,"5min":18.431000000000001,"15min":21.231000000000002},"min":{"1min":6.5739999999999998,"5min":6.5339999999999998,"15min":6.5170000000000003},"max":{"1min":53.826000000000001,"5min":128.06100000000001,"15min":241.69},"last":29.111000000000001}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":20.681999999999999,"5min":18.628,"15min":19.465},"min":{"1min":7.5700000000000003,"5min":7.0599999999999996,"15min":6.5570000000000004},"max":{"1min":69.706000000000003,"5min":130.738,"15min":229.14500000000001},"last":10.497999999999999},{"interface":"front","average":{"1min":24.361999999999998,"5min":22.751999999999999,"15min":21.934999999999999},"min":{"1min":10.942,"5min":10.942,"15min":5.9829999999999997},"max":{"1min":76.808000000000007,"5min":105.361,"15min":284.95100000000002},"last":16.859000000000002}]},{"osd":5,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.340999999999999,"5min":16.795999999999999,"15min":17.544},"min":{"1min":8.1159999999999997,"5min":6.0019999999999998,"15min":5.9489999999999998},"max":{"1min":43.698,"5min":144.864,"15min":247.37799999999999},"last":9.7230000000000008},{"interface":"front","average":{"1min":18.547999999999998,"5min":21.937999999999999,"15min":21.414999999999999},"min":{"1min":10.209,"5min":6.7229999999999999,"15min":6.7229999999999999},"max":{"1min":57.433999999999997,"5min":143.517,"15min":218.82400000000001},"last":11.361000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":22.513000000000002,"5min":23.238,"15min":25.574000000000002},"min":{"1min":9.5549999999999997,"5min":7.7190000000000003,"15min":7.7190000000000003},"max":{"1min":64.820999999999998,"5min":133.203,"15min":288.38299999999998},"last":12.407999999999999},{"interface":"front","average":{"1min":21.914999999999999,"5min":25.452999999999999,"15min":29.690999999999999},"min":{"1min":12.843999999999999,"5min":10.220000000000001,"15min":8.25},"max":{"1min":55.670999999999999,"5min":182.745,"15min":291.435},"last":17.879000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.765999999999998,"5min":19.440000000000001,"15min":23.538},"min":{"1min":7.7519999999999998,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":45.223999999999997,"5min":162.23500000000001,"15min":302.30900000000003},"last":12.992000000000001},{"interface":"front","average":{"1min":22.626000000000001,"5min":25.323,"15min":24.372},"min":{"1min":10.784000000000001,"5min":9.3190000000000008,"15min":5.7859999999999996},"max":{"1min":66.337999999999994,"5min":136.339,"15min":279.00700000000001},"last":14.164}]}]},{"osd":1,"up_from":17,"seq":73014444301,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60360,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311480,"statfs":{"total":96636764160,"available":96574955520,"internally_reserved":0,"allocated":7139328,"data_stored":4075528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":14.016,"5min":26.524000000000001,"15min":23.024000000000001},"min":{"1min":4.9160000000000004,"5min":4.3029999999999999,"15min":4.2080000000000002},"max":{"1min":93.808999999999997,"5min":348.17200000000003,"15min":410.08199999999999},"last":10.736000000000001},{"interface":"front","average":{"1min":15.234999999999999,"5min":20.524999999999999,"15min":20.359000000000002},"min":{"1min":6.4189999999999996,"5min":5.5659999999999998,"15min":5.282},"max":{"1min":82.144999999999996,"5min":146.68700000000001,"15min":303.291},"last":5.843}]},{"osd":2,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":21.712,"15min":22.257999999999999},"min":{"1min":6.3479999999999999,"5min":4.4240000000000004,"15min":4.4240000000000004},"max":{"1min":23.940000000000001,"5min":192.226,"15min":336.255},"last":12.038},{"interface":"front","average":{"1min":18.809999999999999,"5min":28.77,"15min":22.311},"min":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":5.2679999999999998},"max":{"1min":96.281000000000006,"5min":229.56100000000001,"15min":335.88499999999999},"last":14.234999999999999}]},{"osd":3,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.492999999999999,"5min":28.306000000000001,"15min":25.484000000000002},"min":{"1min":9.2159999999999993,"5min":7.0449999999999999,"15min":6.3940000000000001},"max":{"1min":87.390000000000001,"5min":317.94999999999999,"15min":429.30399999999997},"last":6.8810000000000002},{"interface":"front","average":{"1min":14.365,"5min":31.286000000000001,"15min":24.562000000000001},"min":{"1min":7.1550000000000002,"5min":6.3010000000000002,"15min":6.1589999999999998},"max":{"1min":46.652999999999999,"5min":302.00599999999997,"15min":382.79399999999998},"last":8.2799999999999994}]},{"osd":4,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.879999999999999,"5min":30.443999999999999,"15min":29.187000000000001},"min":{"1min":10.609,"5min":8.7759999999999998,"15min":5.9969999999999999},"max":{"1min":83.418000000000006,"5min":207.262,"15min":406.53899999999999},"last":11.395},{"interface":"front","average":{"1min":18.018999999999998,"5min":33.978999999999999,"15min":28.32},"min":{"1min":9.6039999999999992,"5min":7.4039999999999999,"15min":6.9279999999999999},"max":{"1min":96.866,"5min":417.81700000000001,"15min":417.81700000000001},"last":16.530000000000001}]},{"osd":5,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":23.068999999999999,"5min":27.93,"15min":28.184999999999999},"min":{"1min":11.724,"5min":10.911,"15min":7.3049999999999997},"max":{"1min":106.518,"5min":128.53899999999999,"15min":390.11000000000001},"last":13.510999999999999},{"interface":"front","average":{"1min":21.890000000000001,"5min":33.082000000000001,"15min":27.628},"min":{"1min":10.880000000000001,"5min":8.8249999999999993,"15min":7.0350000000000001},"max":{"1min":102.294,"5min":323.339,"15min":357.44799999999998},"last":17.350000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":22.527000000000001,"5min":29.898,"15min":31.405999999999999},"min":{"1min":11.798,"5min":10.257,"15min":7.9279999999999999},"max":{"1min":90.643000000000001,"5min":188.14400000000001,"15min":420.565},"last":14.884},{"interface":"front","average":{"1min":26.419,"5min":35.195999999999998,"15min":32.094999999999999},"min":{"1min":14.733000000000001,"5min":12.470000000000001,"15min":8.8680000000000003},"max":{"1min":94.477999999999994,"5min":264.226,"15min":340.71100000000001},"last":19.036000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":17.436,"5min":24.896999999999998,"15min":27.373999999999999},"min":{"1min":8.2690000000000001,"5min":6.6500000000000004,"15min":6.6500000000000004},"max":{"1min":77.435000000000002,"5min":200.01900000000001,"15min":442.14100000000002},"last":12.789999999999999},{"interface":"front","average":{"1min":24.981000000000002,"5min":32.478999999999999,"15min":30.864000000000001},"min":{"1min":13.936999999999999,"5min":10.494999999999999,"15min":8.8699999999999992},"max":{"1min":98.965999999999994,"5min":267.07999999999998,"15min":533.87199999999996},"last":18.231000000000002}]}]},{"osd":0,"up_from":16,"seq":68719477005,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59968,"kb_used_data":6580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311872,"statfs":{"total":96636764160,"available":96575356928,"internally_reserved":0,"allocated":6737920,"data_stored":3677688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.449,"5min":20.334,"15min":21.199999999999999},"min":{"1min":7.2389999999999999,"5min":5.3540000000000001,"15min":5.3529999999999998},"max":{"1min":52.637,"5min":230.792,"15min":230.792},"last":13.096},{"interface":"front","average":{"1min":17.687999999999999,"5min":22.419,"15min":27.146999999999998},"min":{"1min":8.0589999999999993,"5min":6.9210000000000003,"15min":6.9210000000000003},"max":{"1min":68.528000000000006,"5min":119.28,"15min":289.82400000000001},"last":15.699999999999999}]},{"osd":2,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":15.130000000000001,"5min":24.141999999999999,"15min":23.670999999999999},"min":{"1min":9.3160000000000007,"5min":8.8759999999999994,"15min":7.9080000000000004},"max":{"1min":26.367000000000001,"5min":201.53800000000001,"15min":201.53800000000001},"last":14.922000000000001},{"interface":"front","average":{"1min":9.2829999999999995,"5min":17.088999999999999,"15min":16.57},"min":{"1min":5.8230000000000004,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":29.462,"5min":202.648,"15min":214.94},"last":20.934999999999999}]},{"osd":3,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":22.611000000000001,"15min":27.707000000000001},"min":{"1min":6.1550000000000002,"5min":5.3440000000000003,"15min":5.0899999999999999},"max":{"1min":240.876,"5min":283.43000000000001,"15min":932.75999999999999},"last":9.2149999999999999},{"interface":"front","average":{"1min":20.475000000000001,"5min":26.277999999999999,"15min":27.437000000000001},"min":{"1min":11.481,"5min":8.0809999999999995,"15min":8.0809999999999995},"max":{"1min":78.986999999999995,"5min":256.267,"15min":300.13999999999999},"last":16.376000000000001}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":23.826000000000001,"5min":21.247,"15min":28.013000000000002},"min":{"1min":7.1529999999999996,"5min":6.7930000000000001,"15min":6.4649999999999999},"max":{"1min":153.63200000000001,"5min":153.63200000000001,"15min":1049.731},"last":10.750999999999999},{"interface":"front","average":{"1min":27.728000000000002,"5min":27.978999999999999,"15min":28.177},"min":{"1min":13.824,"5min":12.833,"15min":9.5399999999999991},"max":{"1min":108.58799999999999,"5min":158.40100000000001,"15min":283.43700000000001},"last":17.158999999999999}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":24.460999999999999,"5min":30.645,"15min":28.616},"min":{"1min":10.550000000000001,"5min":10.550000000000001,"15min":5.8300000000000001},"max":{"1min":105.548,"5min":248.34299999999999,"15min":255.328},"last":18.388000000000002},{"interface":"front","average":{"1min":20.219999999999999,"5min":23.016999999999999,"15min":30.934000000000001},"min":{"1min":6.8079999999999998,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":185.125,"5min":264.34899999999999,"15min":1217.117},"last":10.013999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.291,"5min":21.850000000000001,"15min":31.241},"min":{"1min":8.1869999999999994,"5min":7.8789999999999996,"15min":7.0350000000000001},"max":{"1min":28.216999999999999,"5min":154.68199999999999,"15min":1252.499},"last":13.855},{"interface":"front","average":{"1min":20.312000000000001,"5min":30.834,"15min":31.664999999999999},"min":{"1min":14.042,"5min":13.167999999999999,"15min":11.638},"max":{"1min":33.158000000000001,"5min":263.43200000000002,"15min":263.43200000000002},"last":17.785}]},{"osd":7,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":22.698,"5min":32.343000000000004,"15min":32.625999999999998},"min":{"1min":15.154999999999999,"5min":13.768000000000001,"15min":13.212},"max":{"1min":37.085000000000001,"5min":140.50200000000001,"15min":279.38900000000001},"last":19.055},{"interface":"front","average":{"1min":14.250999999999999,"5min":24.257999999999999,"15min":23.280999999999999},"min":{"1min":7.8200000000000003,"5min":7.1459999999999999,"15min":6.2990000000000004},"max":{"1min":27.417000000000002,"5min":262.07299999999998,"15min":262.07299999999998},"last":11.558}]}]},{"osd":6,"up_from":14,"seq":60129542413,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59972,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311868,"statfs":{"total":96636764160,"available":96575352832,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":11.407,"5min":16.442,"15min":18.248999999999999},"min":{"1min":5.4500000000000002,"5min":5.4500000000000002,"15min":4.8579999999999997},"max":{"1min":30.831,"5min":227.49000000000001,"15min":227.49000000000001},"last":6.6740000000000004},{"interface":"front","average":{"1min":19.172999999999998,"5min":22.824000000000002,"15min":24.23},"min":{"1min":6.343,"5min":6.343,"15min":5.9909999999999997},"max":{"1min":80.697000000000003,"5min":228.13399999999999,"15min":229.405},"last":15.725}]},{"osd":1,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":14.680999999999999,"5min":21.905999999999999,"15min":22.215},"min":{"1min":8.3089999999999993,"5min":6.2599999999999998,"15min":5.9390000000000001},"max":{"1min":24.978000000000002,"5min":228.745,"15min":234.59399999999999},"last":16.852},{"interface":"front","average":{"1min":10.353,"5min":18.015999999999998,"15min":17.309000000000001},"min":{"1min":5.702,"5min":4.8090000000000002,"15min":4.6100000000000003},"max":{"1min":18.276,"5min":220.78800000000001,"15min":220.78800000000001},"last":7.9660000000000002}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":11.086,"5min":19.494,"15min":18.353000000000002},"min":{"1min":7.4279999999999999,"5min":5.8680000000000003,"15min":4.4210000000000003},"max":{"1min":30.722999999999999,"5min":156.13499999999999,"15min":215.05699999999999},"last":9.9849999999999994},{"interface":"front","average":{"1min":16.837,"5min":26.483000000000001,"15min":25.437999999999999},"min":{"1min":10.422000000000001,"5min":7.9950000000000001,"15min":7.9950000000000001},"max":{"1min":26.327000000000002,"5min":232.45400000000001,"15min":232.45400000000001},"last":17.975000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":12.009,"5min":25.157,"15min":22.254999999999999},"min":{"1min":6.7590000000000003,"5min":6.024,"15min":6.024},"max":{"1min":26.135999999999999,"5min":248.97900000000001,"15min":248.97900000000001},"last":9.3680000000000003},{"interface":"front","average":{"1min":14.058,"5min":25.068000000000001,"15min":22.209},"min":{"1min":9.5719999999999992,"5min":6.0860000000000003,"15min":6.0860000000000003},"max":{"1min":27.628,"5min":236.92699999999999,"15min":236.92699999999999},"last":13.247999999999999}]},{"osd":4,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":19.373999999999999,"5min":27.763000000000002,"15min":23.887},"min":{"1min":6.3899999999999997,"5min":6.3899999999999997,"15min":6.3090000000000002},"max":{"1min":83.894000000000005,"5min":276.92500000000001,"15min":276.92500000000001},"last":12.034000000000001},{"interface":"front","average":{"1min":20.515999999999998,"5min":27.331,"15min":24.074000000000002},"min":{"1min":8.7449999999999992,"5min":7.258,"15min":6.5700000000000003},"max":{"1min":90.427000000000007,"5min":179.64099999999999,"15min":220.881},"last":14.632999999999999}]},{"osd":5,"last update":"Thu Dec 7 07:54:59 2023","interfaces":[{"interface":"back","average":{"1min":16.332000000000001,"5min":26.169,"15min":26.448},"min":{"1min":7.8570000000000002,"5min":7.8570000000000002,"15min":7.8570000000000002},"max":{"1min":34.262999999999998,"5min":237.53700000000001,"15min":237.53700000000001},"last":11.702999999999999},{"interface":"front","average":{"1min":18.193999999999999,"5min":27.972000000000001,"15min":25.896999999999998},"min":{"1min":11.1,"5min":9.1460000000000008,"15min":8.0099999999999998},"max":{"1min":29.687000000000001,"5min":239.673,"15min":239.673},"last":19.071999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":26.934999999999999,"5min":30.998999999999999,"15min":25.236999999999998},"min":{"1min":10.981999999999999,"5min":8.4890000000000008,"15min":7.3090000000000002},"max":{"1min":107.471,"5min":236.26499999999999,"15min":236.26499999999999},"last":12.686999999999999},{"interface":"front","average":{"1min":28.969999999999999,"5min":31.443999999999999,"15min":28.527999999999999},"min":{"1min":17.210999999999999,"5min":10.239000000000001,"15min":10.239000000000001},"max":{"1min":102.93600000000001,"5min":253.29300000000001,"15min":253.29300000000001},"last":20.169}]}]},{"osd":7,"up_from":13,"seq":55834575119,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78016,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293824,"statfs":{"total":96636764160,"available":96556875776,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":10.605,"5min":11.368,"15min":15.794},"min":{"1min":5.6929999999999996,"5min":4.3899999999999997,"15min":4.29},"max":{"1min":17.640999999999998,"5min":127.59,"15min":214.32599999999999},"last":5.4050000000000002},{"interface":"front","average":{"1min":10.164,"5min":10.188000000000001,"15min":15.063000000000001},"min":{"1min":4.3659999999999997,"5min":4.3659999999999997,"15min":4.3659999999999997},"max":{"1min":20.850000000000001,"5min":47.606000000000002,"15min":199.94499999999999},"last":7.4489999999999998}]},{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":12.381,"5min":16.414000000000001,"15min":19.100000000000001},"min":{"1min":6.3719999999999999,"5min":5.8289999999999997,"15min":5.3849999999999998},"max":{"1min":25.638999999999999,"5min":157.16,"15min":198.512},"last":12.087},{"interface":"front","average":{"1min":12.368,"5min":17.263999999999999,"15min":19.622},"min":{"1min":7.4349999999999996,"5min":5.7270000000000003,"15min":5.6929999999999996},"max":{"1min":18.352,"5min":167.78200000000001,"15min":167.78200000000001},"last":10.782999999999999}]},{"osd":2,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.574,"5min":18.367999999999999,"15min":18.265000000000001},"min":{"1min":7.3330000000000002,"5min":6.0049999999999999,"15min":5.9669999999999996},"max":{"1min":23.154,"5min":282.12099999999998,"15min":282.12099999999998},"last":10.071999999999999},{"interface":"front","average":{"1min":13.243,"5min":17.161000000000001,"15min":19.664000000000001},"min":{"1min":6.9939999999999998,"5min":6.9939999999999998,"15min":6.9939999999999998},"max":{"1min":20.405000000000001,"5min":154.376,"15min":216.28299999999999},"last":17.094000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":14.811999999999999,"5min":16.736999999999998,"15min":20.478999999999999},"min":{"1min":9.3559999999999999,"5min":6.4939999999999998,"15min":6.3410000000000002},"max":{"1min":21.123999999999999,"5min":84.263000000000005,"15min":159.934},"last":8.9480000000000004},{"interface":"front","average":{"1min":15.291,"5min":18.553999999999998,"15min":20.609000000000002},"min":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":6.9409999999999998},"max":{"1min":28.382999999999999,"5min":147.85400000000001,"15min":157.43100000000001},"last":15.944000000000001}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":16.672000000000001,"5min":17.713999999999999,"15min":21.768000000000001},"min":{"1min":9.3829999999999991,"5min":7.2110000000000003,"15min":6.7000000000000002},"max":{"1min":33.926000000000002,"5min":94.227000000000004,"15min":204.71199999999999},"last":17.949999999999999},{"interface":"front","average":{"1min":18.524999999999999,"5min":18.350000000000001,"15min":21.297999999999998},"min":{"1min":9.1869999999999994,"5min":7.7839999999999998,"15min":7.1269999999999998},"max":{"1min":52.731000000000002,"5min":93.356999999999999,"15min":259.23200000000003},"last":14.032}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.744,"5min":21.847000000000001,"15min":25.094000000000001},"min":{"1min":10.131,"5min":8.9830000000000005,"15min":8.9830000000000005},"max":{"1min":92.340999999999994,"5min":189.49000000000001,"15min":239.32599999999999},"last":14.948},{"interface":"front","average":{"1min":15.037000000000001,"5min":18.690999999999999,"15min":23.567},"min":{"1min":8.3529999999999998,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":29.849,"5min":157.60400000000001,"15min":224.91900000000001},"last":12.965}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.780999999999999,"5min":20.637,"15min":24.367999999999999},"min":{"1min":12.939,"5min":9.9390000000000001,"15min":8.2750000000000004},"max":{"1min":49.506999999999998,"5min":66.572999999999993,"15min":192.76300000000001},"last":18.936},{"interface":"front","average":{"1min":20.835000000000001,"5min":23.234999999999999,"15min":29.07},"min":{"1min":13.638999999999999,"5min":13.432,"15min":10.176},"max":{"1min":34.186999999999998,"5min":92.688999999999993,"15min":216.917},"last":19.515999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575118,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82568,"kb_used_data":6968,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289272,"statfs":{"total":96636764160,"available":96552214528,"internally_reserved":0,"allocated":7135232,"data_stored":4075523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":11.474,"5min":12.582000000000001,"15min":22.064},"min":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":4.2409999999999997},"max":{"1min":27.518000000000001,"5min":130.309,"15min":1086.095},"last":10.962},{"interface":"front","average":{"1min":11.550000000000001,"5min":16.629999999999999,"15min":20.43},"min":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"max":{"1min":38.128999999999998,"5min":141.636,"15min":403.21899999999999},"last":5.2149999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:36 2023","interfaces":[{"interface":"back","average":{"1min":20.469000000000001,"5min":15.369,"15min":17.274999999999999},"min":{"1min":6.234,"5min":5.4050000000000002,"15min":4.819},"max":{"1min":130.52500000000001,"5min":130.52500000000001,"15min":310.18299999999999},"last":10.688000000000001},{"interface":"front","average":{"1min":23.145,"5min":17.341000000000001,"15min":20.655000000000001},"min":{"1min":7.8479999999999999,"5min":6.4640000000000004,"15min":4.9290000000000003},"max":{"1min":113.23099999999999,"5min":113.23099999999999,"15min":263.452},"last":12.523999999999999}]},{"osd":2,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.693000000000001,"5min":19.597000000000001,"15min":21.677},"min":{"1min":6.149,"5min":6.149,"15min":6.149},"max":{"1min":70.923000000000002,"5min":142.77699999999999,"15min":250.30600000000001},"last":14.316000000000001},{"interface":"front","average":{"1min":16.704999999999998,"5min":18.449000000000002,"15min":20.291},"min":{"1min":8.2810000000000006,"5min":7.8360000000000003,"15min":6.4059999999999997},"max":{"1min":60.683,"5min":148.40100000000001,"15min":300.988},"last":9.0600000000000005}]},{"osd":3,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":17.314,"5min":22.145,"15min":23.927},"min":{"1min":7.9210000000000003,"5min":7.4370000000000003,"15min":6.9809999999999999},"max":{"1min":55.113999999999997,"5min":143.53899999999999,"15min":305.74900000000002},"last":9.9499999999999993},{"interface":"front","average":{"1min":20.635999999999999,"5min":24.911999999999999,"15min":26.052},"min":{"1min":10.500999999999999,"5min":10.500999999999999,"15min":10.500999999999999},"max":{"1min":71.872,"5min":144.41800000000001,"15min":326.85899999999998},"last":16.387}]},{"osd":4,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":15.747999999999999,"5min":21.678000000000001,"15min":24.283000000000001},"min":{"1min":7.5999999999999996,"5min":6.5899999999999999,"15min":6.5899999999999999},"max":{"1min":68.769999999999996,"5min":136.691,"15min":320.23099999999999},"last":18.332999999999998},{"interface":"front","average":{"1min":17.957000000000001,"5min":24.734000000000002,"15min":23.757999999999999},"min":{"1min":10.146000000000001,"5min":8.7840000000000007,"15min":6.6929999999999996},"max":{"1min":39.585999999999999,"5min":167.351,"15min":313.42500000000001},"last":17.353999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.943999999999999,"5min":25.558,"15min":30.716999999999999},"min":{"1min":10.445,"5min":9.1029999999999998,"15min":8.9280000000000008},"max":{"1min":35.082999999999998,"5min":201.87299999999999,"15min":320.84199999999998},"last":13.159000000000001},{"interface":"front","average":{"1min":18.722000000000001,"5min":25.786999999999999,"15min":30.187999999999999},"min":{"1min":11.132,"5min":8.3030000000000008,"15min":6.3630000000000004},"max":{"1min":32.395000000000003,"5min":192.81800000000001,"15min":365.91899999999998},"last":15.430999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":13.975,"5min":19.853000000000002,"15min":21.242999999999999},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.484},"max":{"1min":24.908999999999999,"5min":203.636,"15min":286.029},"last":19.135000000000002},{"interface":"front","average":{"1min":17.344999999999999,"5min":24.433,"15min":28.132999999999999},"min":{"1min":8.4990000000000006,"5min":8.4990000000000006,"15min":8.4990000000000006},"max":{"1min":31.541,"5min":204.45500000000001,"15min":300.18200000000002},"last":20.856000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575117,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59972,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311868,"statfs":{"total":96636764160,"available":96575352832,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":10.369999999999999,"5min":13.005000000000001,"15min":17.228999999999999},"min":{"1min":4.5499999999999998,"5min":4.5499999999999998,"15min":4.3230000000000004},"max":{"1min":22.457999999999998,"5min":98.765000000000001,"15min":301.959},"last":6.9450000000000003},{"interface":"front","average":{"1min":11.028,"5min":15.567,"15min":21.140999999999998},"min":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"max":{"1min":24.654,"5min":110.508,"15min":363.75099999999998},"last":9.6110000000000007}]},{"osd":1,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":14.859999999999999,"5min":15.579000000000001,"15min":20.225000000000001},"min":{"1min":6.4900000000000002,"5min":6.4900000000000002,"15min":5.577},"max":{"1min":57.762,"5min":121.346,"15min":219.19300000000001},"last":14.085000000000001},{"interface":"front","average":{"1min":13.955,"5min":16.913,"15min":20.986000000000001},"min":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3209999999999997},"max":{"1min":38.786000000000001,"5min":111.107,"15min":183.60400000000001},"last":10.253}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.715999999999999,"5min":21.856000000000002,"15min":20.428999999999998},"min":{"1min":8.4719999999999995,"5min":7.8289999999999997,"15min":6.8710000000000004},"max":{"1min":20.859000000000002,"5min":176.452,"15min":176.452},"last":12.675000000000001},{"interface":"front","average":{"1min":15.119999999999999,"5min":22.337,"15min":21.960000000000001},"min":{"1min":8.2330000000000005,"5min":7.6020000000000003,"15min":7.5940000000000003},"max":{"1min":19.547000000000001,"5min":239.935,"15min":239.935},"last":14.718}]},{"osd":3,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":9.5579999999999998,"5min":18.57,"15min":22.199000000000002},"min":{"1min":5.968,"5min":5.968,"15min":5.968},"max":{"1min":13.237,"5min":177.48699999999999,"15min":246.523},"last":7.984},{"interface":"front","average":{"1min":14.81,"5min":20.649000000000001,"15min":18.181999999999999},"min":{"1min":7.4720000000000004,"5min":6.274,"15min":5.4240000000000004},"max":{"1min":20.555,"5min":183.75399999999999,"15min":183.75399999999999},"last":17.372}]},{"osd":5,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.641,"5min":15.577,"15min":19.568999999999999},"min":{"1min":8.4770000000000003,"5min":6.327,"15min":6.327},"max":{"1min":20.013000000000002,"5min":123.998,"15min":148.21000000000001},"last":11.000999999999999},{"interface":"front","average":{"1min":18.177,"5min":22.536000000000001,"15min":23.007999999999999},"min":{"1min":13.364000000000001,"5min":9.1270000000000007,"15min":6.46},"max":{"1min":25.591000000000001,"5min":224.55500000000001,"15min":224.55500000000001},"last":20.047999999999998}]},{"osd":6,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":13.776999999999999,"5min":20.428000000000001,"15min":23.344000000000001},"min":{"1min":8.4740000000000002,"5min":8.2129999999999992,"15min":7.8079999999999998},"max":{"1min":22.021000000000001,"5min":158.36099999999999,"15min":268.392},"last":13.439},{"interface":"front","average":{"1min":17.134,"5min":25.366,"15min":26.675999999999998},"min":{"1min":13.542999999999999,"5min":10.300000000000001,"15min":9.9299999999999997},"max":{"1min":23.768000000000001,"5min":243.40700000000001,"15min":243.40700000000001},"last":19.326000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.846,"5min":24.731999999999999,"15min":26.576000000000001},"min":{"1min":10.978,"5min":9.9939999999999998,"15min":9.2319999999999993},"max":{"1min":25.713000000000001,"5min":225.375,"15min":261.75099999999998},"last":18.646999999999998},{"interface":"front","average":{"1min":13.435,"5min":19.010999999999999,"15min":19.09},"min":{"1min":9.1319999999999997,"5min":7.181,"15min":6.7110000000000003},"max":{"1min":20.039000000000001,"5min":167.53700000000001,"15min":178.39699999999999},"last":16.396000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.17 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.16 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.15 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.14 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.13 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.11 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.10 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.244 INFO:tasks.ceph:pgid 11.1f last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 11.1e last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 11.1d last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 11.1c last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-07T07:33:40.740374+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 11.e last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-07T07:33:40.740374+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 11.f last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-07T07:33:40.740374+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 11.c last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.245 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-07T07:33:40.740374+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.246 INFO:tasks.ceph:pgid 11.d last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.246 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-07T07:33:40.740374+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.246 INFO:tasks.ceph:pgid 11.b last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.246 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-07T07:33:40.740374+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.246 INFO:tasks.ceph:pgid 11.8 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.246 INFO:tasks.ceph:pgid 11.9 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.246 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-07T07:33:40.740374+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-07T07:33:30.679434+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=33, tm_sec=30, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.a last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.0 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.1 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.2 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.3 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.4 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.5 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.6 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.247 INFO:tasks.ceph:pgid 11.7 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.248 INFO:tasks.ceph:pgid 11.18 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.248 INFO:tasks.ceph:pgid 11.19 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.248 INFO:tasks.ceph:pgid 11.1a last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:55:57.248 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-07T07:56:06.135 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:32.581 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.138 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:32.586 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.152 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:32.601 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.181 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:32.619 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.184 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:32.622 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.185 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:32.634 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.187 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:32.637 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.197 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:32.634 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:06.202 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:32.651 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.114 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:42.563 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.165 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:42.613 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.180 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:42.629 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.212 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:42.661 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.212 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:42.650 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.214 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:42.663 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.214 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:42.652 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.228 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:42.677 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:16.228 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:42.666 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:17.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:56:17.350 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.348+0000 7f59d75ee640 10 monclient: get_monmap_and_config 2023-12-07T07:56:17.350 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.348+0000 7f59d75ee640 10 monclient: build_initial_monmap 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.348+0000 7f59d75ee640 10 monclient: monmap: 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:56:17.349307+0000 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:56:17.349307+0000 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:56:17.351 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:17.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.348+0000 7f59d75ee640 1 Processor -- start 2023-12-07T07:56:17.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 1 -- start start 2023-12-07T07:56:17.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient: init 2023-12-07T07:56:17.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient: _reopen_session rank -1 2023-12-07T07:56:17.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-12-07T07:56:17.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59d00a8910 addr v1:172.21.15.154:6789/0 2023-12-07T07:56:17.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d00a9b70 0x7f59d013f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:17.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient(hunting): picked mon.noname-b con 0x7f59d00a9b70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:56:17.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59d0141f90 addr v1:172.21.15.97:6789/0 2023-12-07T07:56:17.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00a8cf0 con 0x7f59d0141f90 2023-12-07T07:56:17.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00aabd0 con 0x7f59d00a8910 2023-12-07T07:56:17.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:17.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00aaeb0 con 0x7f59d00a9b70 2023-12-07T07:56:17.356 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient(hunting): _renew_subs 2023-12-07T07:56:17.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.349+0000 7f59d75ee640 10 monclient(hunting): authenticate will time out at 2699.728008s 2023-12-07T07:56:17.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.350+0000 7f59d5b8b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f59d0141f90 0x7f59d01423b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:40570/0 (socket says 172.21.15.97:40570) 2023-12-07T07:56:17.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.350+0000 7f59d5b8b640 1 -- 172.21.15.97:0/1272309088 learned_addr learned my addr 172.21.15.97:0/1272309088 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:56:17.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.350+0000 7f59d4b89640 1 --2- 172.21.15.97:0/1272309088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d00a9b70 0x7f59d013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:17.357 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.350+0000 7f59d4b89640 10 monclient(hunting): get_auth_request con 0x7f59d00a9b70 auth_method 0 2023-12-07T07:56:17.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.350+0000 7f59d4b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:17.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.350+0000 7f59d4b89640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:17.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.350+0000 7f59d4b89640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:17.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.351+0000 7f59d4b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:17.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.351+0000 7f59d4b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:17.358 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.351+0000 7f59d4b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:17.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59c7fff640 1 -- 172.21.15.97:0/1272309088 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304743249 0 0) 0x7f59d00a8cf0 con 0x7f59d0141f90 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59c7fff640 10 monclient(hunting): _negotiate 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59c7fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59c7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59c7fff640 10 monclient(hunting): my global_id is 16221 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59c7fff640 1 -- 172.21.15.97:0/1272309088 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59bc003520 con 0x7f59d0141f90 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 10 monclient(hunting): handle_auth_done global_id 17720 payload 274 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 1 -- 172.21.15.97:0/1272309088 >> v1:172.21.15.154:6789/0 conn(0x7f59d00a8910 legacy=0x7f59d013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 1 -- 172.21.15.97:0/1272309088 >> v1:172.21.15.97:6789/0 conn(0x7f59d0141f90 legacy=0x7f59d01423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 10 monclient: _finish_hunting 0 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 1 monclient: found mon.noname-b 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:56:17.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.352+0000 7f59d4b89640 1 -- 172.21.15.97:0/1272309088 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00a9720 con 0x7f59d00a9b70 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.354+0000 7f59d4b89640 1 --2- 172.21.15.97:0/1272309088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d00a9b70 0x7f59d013f370 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f59cc0097d0 tx=0x7f59cc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3475faf889886e71 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 1 -- 172.21.15.97:0/1272309088 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc031890 con 0x7f59d00a9b70 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: dump: 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:17.364 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: _finish_auth 0 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: _check_auth_tickets 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 1 -- 172.21.15.97:0/1272309088 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc0319f0 con 0x7f59d00a9b70 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59d75ee640 5 monclient: authenticate success, global_id 17720 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59d75ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59d75ee640 10 monclient: get_monmap_and_config success 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59d75ee640 4 set_mon_vals no callback set 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 1 -- 172.21.15.97:0/1272309088 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc031d80 con 0x7f59d00a9b70 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.358+0000 7f59c7fff640 10 monclient: dump: 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:17.365 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 10 monclient: shutdown 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 1 -- 172.21.15.97:0/1272309088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d00a9b70 msgr2=0x7f59d013f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 1 --2- 172.21.15.97:0/1272309088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d00a9b70 0x7f59d013f370 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f59cc0097d0 tx=0x7f59cc02b930 comp rx=0 tx=0).stop 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 1 -- 172.21.15.97:0/1272309088 shutdown_connections 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 1 --2- 172.21.15.97:0/1272309088 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d00a9b70 0x7f59d013f370 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 1 -- 172.21.15.97:0/1272309088 >> 172.21.15.97:0/1272309088 conn(0x7f59d00b7780 msgr2=0x7f59d00b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 1 -- 172.21.15.97:0/1272309088 shutdown_connections 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.359+0000 7f59d75ee640 1 -- 172.21.15.97:0/1272309088 wait complete. 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 10 monclient: build_initial_monmap 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 10 monclient: monmap: 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:17.367 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 1 librados: starting msgr at 2023-12-07T07:56:17.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 1 librados: starting objecter 2023-12-07T07:56:17.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 1 Processor -- start 2023-12-07T07:56:17.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 1 -- start start 2023-12-07T07:56:17.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 1 librados: setting wanted keys 2023-12-07T07:56:17.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 1 librados: calling monclient init 2023-12-07T07:56:17.368 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.360+0000 7f59d75ee640 10 monclient: init 2023-12-07T07:56:17.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient: _reopen_session rank -1 2023-12-07T07:56:17.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-07T07:56:17.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d00a8910 0x7f59d013ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:17.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): picked mon.noname-a con 0x7f59d00a8910 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:56:17.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d0141f90 0x7f59d00ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:17.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): picked mon.noname-c con 0x7f59d0141f90 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:56:17.371 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59d0137c40 0x7f59d01380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): picked mon.noname-b con 0x7f59d0137c40 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00ac930 con 0x7f59d00a8910 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00abfd0 con 0x7f59d0137c40 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d013b3e0 con 0x7f59d0141f90 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): _renew_subs 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d75ee640 10 monclient(hunting): authenticate will time out at 2699.739454s 2023-12-07T07:56:17.372 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d538a640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d00a8910 0x7f59d013ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d4b89640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d0141f90 0x7f59d00ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d5b8b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59d0137c40 0x7f59d01380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d538a640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d00a8910 0x7f59d013ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41118/0 (socket says 172.21.15.97:41118) 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d538a640 1 -- 172.21.15.97:0/2617322671 learned_addr learned my addr 172.21.15.97:0/2617322671 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d538a640 10 monclient(hunting): get_auth_request con 0x7f59d00a8910 auth_method 0 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d538a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d538a640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d538a640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d4b89640 10 monclient(hunting): get_auth_request con 0x7f59d0141f90 auth_method 0 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d4b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d4b89640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.361+0000 7f59d4b89640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d5b8b640 10 monclient(hunting): get_auth_request con 0x7f59d0137c40 auth_method 0 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d5b8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d5b8b640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:17.373 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d5b8b640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d538a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d538a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d538a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d4b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d4b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d4b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d5b8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d5b8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.362+0000 7f59d5b8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 10 monclient(hunting): handle_auth_done global_id 16224 payload 824 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 1 -- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d0141f90 msgr2=0x7f59d00ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d0141f90 0x7f59d00ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 1 -- 172.21.15.97:0/2617322671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59d0137c40 msgr2=0x7f59d01380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:17.374 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59d0137c40 0x7f59d01380d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 10 monclient: _finish_hunting 0 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 1 monclient: found mon.noname-a 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d538a640 1 -- 172.21.15.97:0/2617322671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc009400 con 0x7f59d00a8910 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.365+0000 7f59d4b89640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d0141f90 0x7f59d00ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.367+0000 7f59d538a640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d00a8910 0x7f59d013ad50 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7f59c000c400 tx=0x7f59c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.367+0000 7f59c5ffb640 10 client.?.objecter ms_handle_connect 0x7f59d00a8910 2023-12-07T07:56:17.375 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.367+0000 7f59c5ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:56:17.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 1 -- 172.21.15.97:0/2617322671 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00052d0 con 0x7f59d00a8910 2023-12-07T07:56:17.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:17.376 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: dump: 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: _finish_auth 0 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 1 -- 172.21.15.97:0/2617322671 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0005450 con 0x7f59d00a8910 2023-12-07T07:56:17.377 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59d75ee640 5 monclient: authenticate success, global_id 16224 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 1 -- 172.21.15.97:0/2617322671 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0007260 con 0x7f59d00a8910 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c5ffb640 10 monclient: dump: 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:17.378 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 10 monclient: _renew_subs 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d01395e0 con 0x7f59d00a8910 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.370+0000 7f59c77fe640 4 set_mon_vals no callback set 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 10 client.16224.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 10 monclient: _renew_subs 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d013b900 con 0x7f59d00a8910 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.371+0000 7f59d75ee640 1 librados: init done 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.372+0000 7f59d75ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.372+0000 7f59d75ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.372+0000 7f59d75ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:56:17.379 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.372+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d013baf0 con 0x7f59d00a8910 2023-12-07T07:56:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.617+0000 7f59c5ffb640 1 -- 172.21.15.97:0/2617322671 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f59c00073c0 con 0x7f59d00a8910 2023-12-07T07:56:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.617+0000 7f59c5ffb640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f59b406c850 0x7f59b406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.617+0000 7f59d4b89640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f59b406c850 0x7f59b406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.617+0000 7f59d4b89640 10 monclient: get_auth_request con 0x7f59b406c850 auth_method 0 2023-12-07T07:56:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.618+0000 7f59c5ffb640 10 client.16224.objecter ms_handle_connect 0x7f59b406c850 2023-12-07T07:56:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.618+0000 7f59d4b89640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f59b406c850 0x7f59b406ed10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f59cc036000 tx=0x7f59cc03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.630+0000 7f59c5ffb640 1 -- 172.21.15.97:0/2617322671 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f59c0058040 con 0x7f59d00a8910 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.630+0000 7f59c5ffb640 10 client.16224.objecter ms_dispatch 0x7f59d0000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.630+0000 7f59c5ffb640 3 client.16224.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.630+0000 7f59c5ffb640 3 client.16224.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.631+0000 7f59c5ffb640 20 client.16224.objecter dump_active .. 0 homeless 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.631+0000 7f59c5ffb640 1 -- 172.21.15.97:0/2617322671 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59c0054380 con 0x7f59d00a8910 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.631+0000 7f59c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:17.633 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.631+0000 7f59c5ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:56:17.833 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.830+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59d013a390 con 0x7f59b406c850 2023-12-07T07:56:17.837 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.835+0000 7f59c5ffb640 1 -- 172.21.15.97:0/2617322671 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152341 (secure 0 0 0) 0x7f59cc045410 con 0x7f59b406c850 2023-12-07T07:56:17.839 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:56:17.839 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:56:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 10 librados: watch_flush enter 2023-12-07T07:56:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 10 librados: watch_flush exit 2023-12-07T07:56:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 20 client.16224.objecter shutdown clearing up homeless session... 2023-12-07T07:56:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 10 client.16224.objecter successfully canceled tick 2023-12-07T07:56:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f59b406c850 msgr2=0x7f59b406ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f59b406c850 0x7f59b406ed10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f59cc036000 tx=0x7f59cc03c040 comp rx=0 tx=0).stop 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 10 monclient: shutdown 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d00a8910 msgr2=0x7f59d013ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d00a8910 0x7f59d013ad50 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7f59c000c400 tx=0x7f59c000c8c0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 shutdown_connections 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7f59b406c850 0x7f59b406ed10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59d0137c40 0x7f59d01380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f59d0141f90 0x7f59d00ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 --2- 172.21.15.97:0/2617322671 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59d00a8910 0x7f59d013ad50 unknown :-1 s=CLOSED pgs=3244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 >> 172.21.15.97:0/2617322671 conn(0x7f59d00b7780 msgr2=0x7f59d0137190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 shutdown_connections 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 -- 172.21.15.97:0/2617322671 wait complete. 2023-12-07T07:56:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:17.837+0000 7f59d75ee640 1 librados: shutdown 2023-12-07T07:56:17.850 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2023-12-07T07:56:15.969573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542936,"kb_used_data":53400,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754431784,"statfs":{"total":773094113280,"available":772538146816,"internally_reserved":0,"allocated":54681600,"data_stored":30217154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":501206576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"11.17","version":"0'0","reported_seq":873,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:05.576361+0000","last_change":"2023-12-07T07:56:05.577005+0000","last_active":"2023-12-07T07:56:05.576361+0000","last_peered":"2023-12-07T07:56:05.576361+0000","last_clean":"2023-12-07T07:56:05.576361+0000","last_became_active":"2023-12-07T07:34:49.221580+0000","last_became_peered":"2023-12-07T07:34:49.221580+0000","last_unstale":"2023-12-07T07:56:05.576361+0000","last_undegraded":"2023-12-07T07:56:05.576361+0000","last_fullsized":"2023-12-07T07:56:05.576361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:05.575074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:05.575074+0000","last_clean_scrub_stamp":"2023-12-07T07:56:05.575074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:51:34.125625+0000","scrub_duration":0.070435624000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:07.470197+0000","last_change":"2023-12-07T07:56:07.470832+0000","last_active":"2023-12-07T07:56:07.470197+0000","last_peered":"2023-12-07T07:56:07.470197+0000","last_clean":"2023-12-07T07:56:07.470197+0000","last_became_active":"2023-12-07T07:34:47.563950+0000","last_became_peered":"2023-12-07T07:34:47.563950+0000","last_unstale":"2023-12-07T07:56:07.470197+0000","last_undegraded":"2023-12-07T07:56:07.470197+0000","last_fullsized":"2023-12-07T07:56:07.470197+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:07.468907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:07.468907+0000","last_clean_scrub_stamp":"2023-12-07T07:56:07.468907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:40.285890+0000","scrub_duration":0.023397939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":887,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:59.286845+0000","last_change":"2023-12-07T07:55:59.287565+0000","last_active":"2023-12-07T07:55:59.286845+0000","last_peered":"2023-12-07T07:55:59.286845+0000","last_clean":"2023-12-07T07:55:59.286845+0000","last_became_active":"2023-12-07T07:34:47.035305+0000","last_became_peered":"2023-12-07T07:34:47.035305+0000","last_unstale":"2023-12-07T07:55:59.286845+0000","last_undegraded":"2023-12-07T07:55:59.286845+0000","last_fullsized":"2023-12-07T07:55:59.286845+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:59.285437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:59.285437+0000","last_clean_scrub_stamp":"2023-12-07T07:55:59.285437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:24:06.124323+0000","scrub_duration":0.056762805999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":883,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:49.874362+0000","last_change":"2023-12-07T07:34:55.054864+0000","last_active":"2023-12-07T07:55:49.874362+0000","last_peered":"2023-12-07T07:55:49.874362+0000","last_clean":"2023-12-07T07:55:49.874362+0000","last_became_active":"2023-12-07T07:34:55.049584+0000","last_became_peered":"2023-12-07T07:34:55.049584+0000","last_unstale":"2023-12-07T07:55:49.874362+0000","last_undegraded":"2023-12-07T07:55:49.874362+0000","last_fullsized":"2023-12-07T07:55:49.874362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":875,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:53.431731+0000","last_change":"2023-12-07T07:55:53.432421+0000","last_active":"2023-12-07T07:55:53.431731+0000","last_peered":"2023-12-07T07:55:53.431731+0000","last_clean":"2023-12-07T07:55:53.431731+0000","last_became_active":"2023-12-07T07:35:04.480195+0000","last_became_peered":"2023-12-07T07:35:04.480195+0000","last_unstale":"2023-12-07T07:55:53.431731+0000","last_undegraded":"2023-12-07T07:55:53.431731+0000","last_fullsized":"2023-12-07T07:55:53.431731+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:53.430345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:53.430345+0000","last_clean_scrub_stamp":"2023-12-07T07:55:53.430345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:20:02.175037+0000","scrub_duration":0.080473749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":878,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:54.451518+0000","last_change":"2023-12-07T07:55:54.452220+0000","last_active":"2023-12-07T07:55:54.451518+0000","last_peered":"2023-12-07T07:55:54.451518+0000","last_clean":"2023-12-07T07:55:54.451518+0000","last_became_active":"2023-12-07T07:34:51.735923+0000","last_became_peered":"2023-12-07T07:34:51.735923+0000","last_unstale":"2023-12-07T07:55:54.451518+0000","last_undegraded":"2023-12-07T07:55:54.451518+0000","last_fullsized":"2023-12-07T07:55:54.451518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:54.447971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:54.447971+0000","last_clean_scrub_stamp":"2023-12-07T07:55:54.447971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:00:50.135085+0000","scrub_duration":0.041230276000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":900,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:06.482209+0000","last_change":"2023-12-07T07:56:06.482907+0000","last_active":"2023-12-07T07:56:06.482209+0000","last_peered":"2023-12-07T07:56:06.482209+0000","last_clean":"2023-12-07T07:56:06.482209+0000","last_became_active":"2023-12-07T07:34:47.052848+0000","last_became_peered":"2023-12-07T07:34:47.052848+0000","last_unstale":"2023-12-07T07:56:06.482209+0000","last_undegraded":"2023-12-07T07:56:06.482209+0000","last_fullsized":"2023-12-07T07:56:06.482209+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:06.480815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:06.480815+0000","last_clean_scrub_stamp":"2023-12-07T07:56:06.480815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:04:01.002958+0000","scrub_duration":0.027502794000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:03.309171+0000","last_change":"2023-12-07T07:56:03.309848+0000","last_active":"2023-12-07T07:56:03.309171+0000","last_peered":"2023-12-07T07:56:03.309171+0000","last_clean":"2023-12-07T07:56:03.309171+0000","last_became_active":"2023-12-07T07:34:51.115413+0000","last_became_peered":"2023-12-07T07:34:51.115413+0000","last_unstale":"2023-12-07T07:56:03.309171+0000","last_undegraded":"2023-12-07T07:56:03.309171+0000","last_fullsized":"2023-12-07T07:56:03.309171+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:03.307721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:03.307721+0000","last_clean_scrub_stamp":"2023-12-07T07:56:03.307721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:09:42.334141+0000","scrub_duration":0.033872505999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":901,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:10.553165+0000","last_change":"2023-12-07T07:56:10.553838+0000","last_active":"2023-12-07T07:56:10.553165+0000","last_peered":"2023-12-07T07:56:10.553165+0000","last_clean":"2023-12-07T07:56:10.553165+0000","last_became_active":"2023-12-07T07:34:47.149617+0000","last_became_peered":"2023-12-07T07:34:47.149617+0000","last_unstale":"2023-12-07T07:56:10.553165+0000","last_undegraded":"2023-12-07T07:56:10.553165+0000","last_fullsized":"2023-12-07T07:56:10.553165+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:10.551789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:10.551789+0000","last_clean_scrub_stamp":"2023-12-07T07:56:10.551789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:28:24.694774+0000","scrub_duration":0.066138038999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":884,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.582790+0000","last_change":"2023-12-07T07:34:53.278594+0000","last_active":"2023-12-07T07:55:46.582790+0000","last_peered":"2023-12-07T07:55:46.582790+0000","last_clean":"2023-12-07T07:55:46.582790+0000","last_became_active":"2023-12-07T07:34:53.269724+0000","last_became_peered":"2023-12-07T07:34:53.269724+0000","last_unstale":"2023-12-07T07:55:46.582790+0000","last_undegraded":"2023-12-07T07:55:46.582790+0000","last_fullsized":"2023-12-07T07:55:46.582790+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:05.498663+0000","last_change":"2023-12-07T07:56:05.499436+0000","last_active":"2023-12-07T07:56:05.498663+0000","last_peered":"2023-12-07T07:56:05.498663+0000","last_clean":"2023-12-07T07:56:05.498663+0000","last_became_active":"2023-12-07T07:34:56.746142+0000","last_became_peered":"2023-12-07T07:34:56.746142+0000","last_unstale":"2023-12-07T07:56:05.498663+0000","last_undegraded":"2023-12-07T07:56:05.498663+0000","last_fullsized":"2023-12-07T07:56:05.498663+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:05.497262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:05.497262+0000","last_clean_scrub_stamp":"2023-12-07T07:56:05.497262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:09:06.700261+0000","scrub_duration":0.044731305999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":877,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:59.357476+0000","last_change":"2023-12-07T07:55:59.358134+0000","last_active":"2023-12-07T07:55:59.357476+0000","last_peered":"2023-12-07T07:55:59.357476+0000","last_clean":"2023-12-07T07:55:59.357476+0000","last_became_active":"2023-12-07T07:34:49.356256+0000","last_became_peered":"2023-12-07T07:34:49.356256+0000","last_unstale":"2023-12-07T07:55:59.357476+0000","last_undegraded":"2023-12-07T07:55:59.357476+0000","last_fullsized":"2023-12-07T07:55:59.357476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:59.356120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:59.356120+0000","last_clean_scrub_stamp":"2023-12-07T07:55:59.356120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:29:30.403784+0000","scrub_duration":0.028239416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:05.561765+0000","last_change":"2023-12-07T07:56:05.562470+0000","last_active":"2023-12-07T07:56:05.561765+0000","last_peered":"2023-12-07T07:56:05.561765+0000","last_clean":"2023-12-07T07:56:05.561765+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:56:05.561765+0000","last_undegraded":"2023-12-07T07:56:05.561765+0000","last_fullsized":"2023-12-07T07:56:05.561765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:05.558570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:05.558570+0000","last_clean_scrub_stamp":"2023-12-07T07:56:05.558570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:46:32.125782+0000","scrub_duration":0.058343892000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:56.535152+0000","last_change":"2023-12-07T07:55:56.536009+0000","last_active":"2023-12-07T07:55:56.535152+0000","last_peered":"2023-12-07T07:55:56.535152+0000","last_clean":"2023-12-07T07:55:56.535152+0000","last_became_active":"2023-12-07T07:34:48.171478+0000","last_became_peered":"2023-12-07T07:34:48.171478+0000","last_unstale":"2023-12-07T07:55:56.535152+0000","last_undegraded":"2023-12-07T07:55:56.535152+0000","last_fullsized":"2023-12-07T07:55:56.535152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:56.533551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:56.533551+0000","last_clean_scrub_stamp":"2023-12-07T07:55:56.533551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:40:56.809166+0000","scrub_duration":0.044084212999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":887,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:56.747016+0000","last_change":"2023-12-07T07:55:56.747748+0000","last_active":"2023-12-07T07:55:56.747016+0000","last_peered":"2023-12-07T07:55:56.747016+0000","last_clean":"2023-12-07T07:55:56.747016+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:55:56.747016+0000","last_undegraded":"2023-12-07T07:55:56.747016+0000","last_fullsized":"2023-12-07T07:55:56.747016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:56.745587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:56.745587+0000","last_clean_scrub_stamp":"2023-12-07T07:55:56.745587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:42:40.818863+0000","scrub_duration":0.035195790999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:01.640440+0000","last_change":"2023-12-07T07:56:01.641084+0000","last_active":"2023-12-07T07:56:01.640440+0000","last_peered":"2023-12-07T07:56:01.640440+0000","last_clean":"2023-12-07T07:56:01.640440+0000","last_became_active":"2023-12-07T07:34:49.980936+0000","last_became_peered":"2023-12-07T07:34:49.980936+0000","last_unstale":"2023-12-07T07:56:01.640440+0000","last_undegraded":"2023-12-07T07:56:01.640440+0000","last_fullsized":"2023-12-07T07:56:01.640440+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:01.637388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:01.637388+0000","last_clean_scrub_stamp":"2023-12-07T07:56:01.637388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:48:08.865963+0000","scrub_duration":0.059076113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:07.537567+0000","last_change":"2023-12-07T07:56:07.538245+0000","last_active":"2023-12-07T07:56:07.537567+0000","last_peered":"2023-12-07T07:56:07.537567+0000","last_clean":"2023-12-07T07:56:07.537567+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:56:07.537567+0000","last_undegraded":"2023-12-07T07:56:07.537567+0000","last_fullsized":"2023-12-07T07:56:07.537567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:07.536194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:07.536194+0000","last_clean_scrub_stamp":"2023-12-07T07:56:07.536194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:42:38.367064+0000","scrub_duration":0.026414302000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":900,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.521742+0000","last_change":"2023-12-07T07:56:09.522424+0000","last_active":"2023-12-07T07:56:09.521742+0000","last_peered":"2023-12-07T07:56:09.521742+0000","last_clean":"2023-12-07T07:56:09.521742+0000","last_became_active":"2023-12-07T07:34:51.754214+0000","last_became_peered":"2023-12-07T07:34:51.754214+0000","last_unstale":"2023-12-07T07:56:09.521742+0000","last_undegraded":"2023-12-07T07:56:09.521742+0000","last_fullsized":"2023-12-07T07:56:09.521742+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.520400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.520400+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.520400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:45:14.583169+0000","scrub_duration":0.029549394999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":893,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:55.267024+0000","last_change":"2023-12-07T07:55:55.267692+0000","last_active":"2023-12-07T07:55:55.267024+0000","last_peered":"2023-12-07T07:55:55.267024+0000","last_clean":"2023-12-07T07:55:55.267024+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:55:55.267024+0000","last_undegraded":"2023-12-07T07:55:55.267024+0000","last_fullsized":"2023-12-07T07:55:55.267024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:55.265638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:55.265638+0000","last_clean_scrub_stamp":"2023-12-07T07:55:55.265638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:51:23.724769+0000","scrub_duration":0.024987489000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":900,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:08.581258+0000","last_change":"2023-12-07T07:56:08.581957+0000","last_active":"2023-12-07T07:56:08.581258+0000","last_peered":"2023-12-07T07:56:08.581258+0000","last_clean":"2023-12-07T07:56:08.581258+0000","last_became_active":"2023-12-07T07:35:04.566145+0000","last_became_peered":"2023-12-07T07:35:04.566145+0000","last_unstale":"2023-12-07T07:56:08.581258+0000","last_undegraded":"2023-12-07T07:56:08.581258+0000","last_fullsized":"2023-12-07T07:56:08.581258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:08.579900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:08.579900+0000","last_clean_scrub_stamp":"2023-12-07T07:56:08.579900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:16:11.983753+0000","scrub_duration":0.053430152000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:00.437553+0000","last_change":"2023-12-07T07:56:00.438194+0000","last_active":"2023-12-07T07:56:00.437553+0000","last_peered":"2023-12-07T07:56:00.437553+0000","last_clean":"2023-12-07T07:56:00.437553+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:56:00.437553+0000","last_undegraded":"2023-12-07T07:56:00.437553+0000","last_fullsized":"2023-12-07T07:56:00.437553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:00.436260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:00.436260+0000","last_clean_scrub_stamp":"2023-12-07T07:56:00.436260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:50:56.094814+0000","scrub_duration":0.026847559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":881,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:07.669052+0000","last_change":"2023-12-07T07:56:07.669763+0000","last_active":"2023-12-07T07:56:07.669052+0000","last_peered":"2023-12-07T07:56:07.669052+0000","last_clean":"2023-12-07T07:56:07.669052+0000","last_became_active":"2023-12-07T07:34:47.437980+0000","last_became_peered":"2023-12-07T07:34:47.437980+0000","last_unstale":"2023-12-07T07:56:07.669052+0000","last_undegraded":"2023-12-07T07:56:07.669052+0000","last_fullsized":"2023-12-07T07:56:07.669052+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:07.667590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:07.667590+0000","last_clean_scrub_stamp":"2023-12-07T07:56:07.667590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:49:49.966944+0000","scrub_duration":0.066626558000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":905,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:04.632372+0000","last_change":"2023-12-07T07:56:04.636635+0000","last_active":"2023-12-07T07:56:04.632372+0000","last_peered":"2023-12-07T07:56:04.632372+0000","last_clean":"2023-12-07T07:56:04.632372+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:56:04.632372+0000","last_undegraded":"2023-12-07T07:56:04.632372+0000","last_fullsized":"2023-12-07T07:56:04.632372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:04.628663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:04.628663+0000","last_clean_scrub_stamp":"2023-12-07T07:56:04.628663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:14:45.191025+0000","scrub_duration":0.171417181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":881,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:58.587036+0000","last_change":"2023-12-07T07:55:58.587733+0000","last_active":"2023-12-07T07:55:58.587036+0000","last_peered":"2023-12-07T07:55:58.587036+0000","last_clean":"2023-12-07T07:55:58.587036+0000","last_became_active":"2023-12-07T07:35:04.015545+0000","last_became_peered":"2023-12-07T07:35:04.015545+0000","last_unstale":"2023-12-07T07:55:58.587036+0000","last_undegraded":"2023-12-07T07:55:58.587036+0000","last_fullsized":"2023-12-07T07:55:58.587036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:58.585592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:58.585592+0000","last_clean_scrub_stamp":"2023-12-07T07:55:58.585592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:15:12.758689+0000","scrub_duration":0.02727336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":897,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:53.453695+0000","last_change":"2023-12-07T07:55:53.453695+0000","last_active":"2023-12-07T07:55:53.453695+0000","last_peered":"2023-12-07T07:55:53.453695+0000","last_clean":"2023-12-07T07:55:53.453695+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:55:53.453695+0000","last_undegraded":"2023-12-07T07:55:53.453695+0000","last_fullsized":"2023-12-07T07:55:53.453695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:53.451589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:53.451589+0000","last_clean_scrub_stamp":"2023-12-07T07:55:53.451589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:55:36.310408+0000","scrub_duration":0.050438442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":885,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:58.796353+0000","last_change":"2023-12-07T07:55:58.797279+0000","last_active":"2023-12-07T07:55:58.796353+0000","last_peered":"2023-12-07T07:55:58.796353+0000","last_clean":"2023-12-07T07:55:58.796353+0000","last_became_active":"2023-12-07T07:34:52.469358+0000","last_became_peered":"2023-12-07T07:34:52.469358+0000","last_unstale":"2023-12-07T07:55:58.796353+0000","last_undegraded":"2023-12-07T07:55:58.796353+0000","last_fullsized":"2023-12-07T07:55:58.796353+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:58.794978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:58.794978+0000","last_clean_scrub_stamp":"2023-12-07T07:55:58.794978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:46:26.469454+0000","scrub_duration":0.034781629000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:04.533048+0000","last_change":"2023-12-07T07:56:04.533683+0000","last_active":"2023-12-07T07:56:04.533048+0000","last_peered":"2023-12-07T07:56:04.533048+0000","last_clean":"2023-12-07T07:56:04.533048+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:56:04.533048+0000","last_undegraded":"2023-12-07T07:56:04.533048+0000","last_fullsized":"2023-12-07T07:56:04.533048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:04.531736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:04.531736+0000","last_clean_scrub_stamp":"2023-12-07T07:56:04.531736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:34:18.839494+0000","scrub_duration":0.025183835000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"920'78","reported_seq":989,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:52.475342+0000","last_change":"2023-12-07T07:55:52.476595+0000","last_active":"2023-12-07T07:55:52.475342+0000","last_peered":"2023-12-07T07:55:52.475342+0000","last_clean":"2023-12-07T07:55:52.475342+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:55:52.475342+0000","last_undegraded":"2023-12-07T07:55:52.475342+0000","last_fullsized":"2023-12-07T07:55:52.475342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'78","last_scrub_stamp":"2023-12-07T07:55:52.472086+0000","last_deep_scrub":"920'78","last_deep_scrub_stamp":"2023-12-07T07:55:52.472086+0000","last_clean_scrub_stamp":"2023-12-07T07:55:52.472086+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:50:00.549127+0000","scrub_duration":0.112648125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.535146+0000","last_change":"2023-12-07T07:56:09.536082+0000","last_active":"2023-12-07T07:56:09.535146+0000","last_peered":"2023-12-07T07:56:09.535146+0000","last_clean":"2023-12-07T07:56:09.535146+0000","last_became_active":"2023-12-07T07:34:47.010508+0000","last_became_peered":"2023-12-07T07:34:47.010508+0000","last_unstale":"2023-12-07T07:56:09.535146+0000","last_undegraded":"2023-12-07T07:56:09.535146+0000","last_fullsized":"2023-12-07T07:56:09.535146+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.530547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.530547+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.530547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:15:04.939748+0000","scrub_duration":0.051766765999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":882,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:08.616961+0000","last_change":"2023-12-07T07:56:08.617643+0000","last_active":"2023-12-07T07:56:08.616961+0000","last_peered":"2023-12-07T07:56:08.616961+0000","last_clean":"2023-12-07T07:56:08.616961+0000","last_became_active":"2023-12-07T07:35:02.498406+0000","last_became_peered":"2023-12-07T07:35:02.498406+0000","last_unstale":"2023-12-07T07:56:08.616961+0000","last_undegraded":"2023-12-07T07:56:08.616961+0000","last_fullsized":"2023-12-07T07:56:08.616961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:08.615636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:08.615636+0000","last_clean_scrub_stamp":"2023-12-07T07:56:08.615636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:23:37.072159+0000","scrub_duration":0.022749836999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1","version":"920'8","reported_seq":890,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:54.579937+0000","last_change":"2023-12-07T07:55:54.581079+0000","last_active":"2023-12-07T07:55:54.579937+0000","last_peered":"2023-12-07T07:55:54.579937+0000","last_clean":"2023-12-07T07:55:54.579937+0000","last_became_active":"2023-12-07T07:34:48.562400+0000","last_became_peered":"2023-12-07T07:34:48.562400+0000","last_unstale":"2023-12-07T07:55:54.579937+0000","last_undegraded":"2023-12-07T07:55:54.579937+0000","last_fullsized":"2023-12-07T07:55:54.579937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'8","last_scrub_stamp":"2023-12-07T07:55:54.577320+0000","last_deep_scrub":"920'8","last_deep_scrub_stamp":"2023-12-07T07:55:54.577320+0000","last_clean_scrub_stamp":"2023-12-07T07:55:54.577320+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:40:35.663149+0000","scrub_duration":0.087017894999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":889,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:11.531462+0000","last_change":"2023-12-07T07:56:11.532133+0000","last_active":"2023-12-07T07:56:11.531462+0000","last_peered":"2023-12-07T07:56:11.531462+0000","last_clean":"2023-12-07T07:56:11.531462+0000","last_became_active":"2023-12-07T07:34:50.845529+0000","last_became_peered":"2023-12-07T07:34:50.845529+0000","last_unstale":"2023-12-07T07:56:11.531462+0000","last_undegraded":"2023-12-07T07:56:11.531462+0000","last_fullsized":"2023-12-07T07:56:11.531462+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:11.529980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:11.529980+0000","last_clean_scrub_stamp":"2023-12-07T07:56:11.529980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:15:52.318349+0000","scrub_duration":0.033477287000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":892,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.608617+0000","last_change":"2023-12-07T07:56:09.609295+0000","last_active":"2023-12-07T07:56:09.608617+0000","last_peered":"2023-12-07T07:56:09.608617+0000","last_clean":"2023-12-07T07:56:09.608617+0000","last_became_active":"2023-12-07T07:34:44.899627+0000","last_became_peered":"2023-12-07T07:34:44.899627+0000","last_unstale":"2023-12-07T07:56:09.608617+0000","last_undegraded":"2023-12-07T07:56:09.608617+0000","last_fullsized":"2023-12-07T07:56:09.608617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.605252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.605252+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.605252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:30:15.455205+0000","scrub_duration":0.053693215000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":876,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:58.522399+0000","last_change":"2023-12-07T07:55:58.523101+0000","last_active":"2023-12-07T07:55:58.522399+0000","last_peered":"2023-12-07T07:55:58.522399+0000","last_clean":"2023-12-07T07:55:58.522399+0000","last_became_active":"2023-12-07T07:34:50.418281+0000","last_became_peered":"2023-12-07T07:34:50.418281+0000","last_unstale":"2023-12-07T07:55:58.522399+0000","last_undegraded":"2023-12-07T07:55:58.522399+0000","last_fullsized":"2023-12-07T07:55:58.522399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:58.518343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:58.518343+0000","last_clean_scrub_stamp":"2023-12-07T07:55:58.518343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:41:54.911463+0000","scrub_duration":0.024849433000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":876,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:54.356961+0000","last_change":"2023-12-07T07:55:54.357655+0000","last_active":"2023-12-07T07:55:54.356961+0000","last_peered":"2023-12-07T07:55:54.356961+0000","last_clean":"2023-12-07T07:55:54.356961+0000","last_became_active":"2023-12-07T07:34:58.263210+0000","last_became_peered":"2023-12-07T07:34:58.263210+0000","last_unstale":"2023-12-07T07:55:54.356961+0000","last_undegraded":"2023-12-07T07:55:54.356961+0000","last_fullsized":"2023-12-07T07:55:54.356961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:54.355609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:54.355609+0000","last_clean_scrub_stamp":"2023-12-07T07:55:54.355609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T16:31:59.519974+0000","scrub_duration":0.046963756000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":880,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:46.593674+0000","last_change":"2023-12-07T07:34:47.231473+0000","last_active":"2023-12-07T07:55:46.593674+0000","last_peered":"2023-12-07T07:55:46.593674+0000","last_clean":"2023-12-07T07:55:46.593674+0000","last_became_active":"2023-12-07T07:34:47.225369+0000","last_became_peered":"2023-12-07T07:34:47.225369+0000","last_unstale":"2023-12-07T07:55:46.593674+0000","last_undegraded":"2023-12-07T07:55:46.593674+0000","last_fullsized":"2023-12-07T07:55:46.593674+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:34:40.816370+0000","last_clean_scrub_stamp":"2023-12-07T07:34:40.816370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":875,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.472149+0000","last_change":"2023-12-07T07:56:09.472800+0000","last_active":"2023-12-07T07:56:09.472149+0000","last_peered":"2023-12-07T07:56:09.472149+0000","last_clean":"2023-12-07T07:56:09.472149+0000","last_became_active":"2023-12-07T07:35:00.363892+0000","last_became_peered":"2023-12-07T07:35:00.363892+0000","last_unstale":"2023-12-07T07:56:09.472149+0000","last_undegraded":"2023-12-07T07:56:09.472149+0000","last_fullsized":"2023-12-07T07:56:09.472149+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.470782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.470782+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.470782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:16:57.692809+0000","scrub_duration":0.063907027000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":880,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:57.547853+0000","last_change":"2023-12-07T07:55:57.548535+0000","last_active":"2023-12-07T07:55:57.547853+0000","last_peered":"2023-12-07T07:55:57.547853+0000","last_clean":"2023-12-07T07:55:57.547853+0000","last_became_active":"2023-12-07T07:34:45.533595+0000","last_became_peered":"2023-12-07T07:34:45.533595+0000","last_unstale":"2023-12-07T07:55:57.547853+0000","last_undegraded":"2023-12-07T07:55:57.547853+0000","last_fullsized":"2023-12-07T07:55:57.547853+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:57.546560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:57.546560+0000","last_clean_scrub_stamp":"2023-12-07T07:55:57.546560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:01:03.889741+0000","scrub_duration":0.024617533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":884,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:57.795572+0000","last_change":"2023-12-07T07:55:57.796234+0000","last_active":"2023-12-07T07:55:57.795572+0000","last_peered":"2023-12-07T07:55:57.795572+0000","last_clean":"2023-12-07T07:55:57.795572+0000","last_became_active":"2023-12-07T07:34:54.918404+0000","last_became_peered":"2023-12-07T07:34:54.918404+0000","last_unstale":"2023-12-07T07:55:57.795572+0000","last_undegraded":"2023-12-07T07:55:57.795572+0000","last_fullsized":"2023-12-07T07:55:57.795572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:57.794282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:57.794282+0000","last_clean_scrub_stamp":"2023-12-07T07:55:57.794282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:32:29.043381+0000","scrub_duration":0.041595646999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":890,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:13.545920+0000","last_change":"2023-12-07T07:56:13.546610+0000","last_active":"2023-12-07T07:56:13.545920+0000","last_peered":"2023-12-07T07:56:13.545920+0000","last_clean":"2023-12-07T07:56:13.545920+0000","last_became_active":"2023-12-07T07:34:45.408105+0000","last_became_peered":"2023-12-07T07:34:45.408105+0000","last_unstale":"2023-12-07T07:56:13.545920+0000","last_undegraded":"2023-12-07T07:56:13.545920+0000","last_fullsized":"2023-12-07T07:56:13.545920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:13.544607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:13.544607+0000","last_clean_scrub_stamp":"2023-12-07T07:56:13.544607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:49:50.853888+0000","scrub_duration":0.026673565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":877,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:55.492980+0000","last_change":"2023-12-07T07:55:55.493639+0000","last_active":"2023-12-07T07:55:55.492980+0000","last_peered":"2023-12-07T07:55:55.492980+0000","last_clean":"2023-12-07T07:55:55.492980+0000","last_became_active":"2023-12-07T07:34:49.378907+0000","last_became_peered":"2023-12-07T07:34:49.378907+0000","last_unstale":"2023-12-07T07:55:55.492980+0000","last_undegraded":"2023-12-07T07:55:55.492980+0000","last_fullsized":"2023-12-07T07:55:55.492980+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:55.491658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:55.491658+0000","last_clean_scrub_stamp":"2023-12-07T07:55:55.491658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:28:12.211489+0000","scrub_duration":0.058683120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444305,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":25.919,"15min":28.937999999999999},"min":{"1min":6.7430000000000003,"5min":5.7389999999999999,"15min":3.8220000000000001},"max":{"1min":57.213999999999999,"5min":240.48099999999999,"15min":963.553},"last":9.7880000000000003},{"interface":"front","average":{"1min":21.895,"5min":32.722000000000001,"15min":26.271999999999998},"min":{"1min":8.7810000000000006,"5min":8.7810000000000006,"15min":4.7640000000000002},"max":{"1min":123.477,"5min":252.37899999999999,"15min":285.09399999999999},"last":12.394}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":9.9960000000000004,"5min":20.056000000000001,"15min":19.806999999999999},"min":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":5.5049999999999999},"max":{"1min":35.368000000000002,"5min":236.608,"15min":240.595},"last":7.5919999999999996},{"interface":"front","average":{"1min":15.894,"5min":28.577000000000002,"15min":25.738},"min":{"1min":9.3689999999999998,"5min":9.0139999999999993,"15min":6.3490000000000002},"max":{"1min":33.170000000000002,"5min":275.33300000000003,"15min":307.64100000000002},"last":11.776}]},{"osd":2,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.742000000000001,"5min":29.213999999999999,"15min":23.890999999999998},"min":{"1min":10.449,"5min":9.9049999999999994,"15min":7.4749999999999996},"max":{"1min":56.265000000000001,"5min":277.43099999999998,"15min":277.43099999999998},"last":13.023},{"interface":"front","average":{"1min":12.913,"5min":22.611999999999998,"15min":24.943000000000001},"min":{"1min":6.3700000000000001,"5min":6.3700000000000001,"15min":6.3700000000000001},"max":{"1min":51.895000000000003,"5min":243.97800000000001,"15min":243.97800000000001},"last":9.3490000000000002}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.201000000000001,"5min":21.512,"15min":22.870000000000001},"min":{"1min":5.7670000000000003,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":37.640999999999998,"5min":195.59299999999999,"15min":314.84300000000002},"last":6.5490000000000004},{"interface":"front","average":{"1min":18.228000000000002,"5min":27.143000000000001,"15min":26.853999999999999},"min":{"1min":9.6170000000000009,"5min":7.1500000000000004,"15min":7.1500000000000004},"max":{"1min":66.102999999999994,"5min":215.51900000000001,"15min":252.49799999999999},"last":15.295}]},{"osd":5,"last update":"Thu Dec 7 07:55:22 2023","interfaces":[{"interface":"back","average":{"1min":19.082999999999998,"5min":28.425999999999998,"15min":28.632999999999999},"min":{"1min":7.2640000000000002,"5min":7.2640000000000002,"15min":7.2640000000000002},"max":{"1min":76.856999999999999,"5min":241.82300000000001,"15min":241.82300000000001},"last":14.618},{"interface":"front","average":{"1min":20.713000000000001,"5min":30.940000000000001,"15min":30.433},"min":{"1min":9.3460000000000001,"5min":9.3460000000000001,"15min":8.6780000000000008},"max":{"1min":57.811,"5min":286.17200000000003,"15min":286.17200000000003},"last":15.951000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.530999999999999,"5min":21.175999999999998,"15min":26.318000000000001},"min":{"1min":7.0590000000000002,"5min":6.149,"15min":6.149},"max":{"1min":84.808000000000007,"5min":168.52600000000001,"15min":333.97000000000003},"last":10.458},{"interface":"front","average":{"1min":19.992000000000001,"5min":28.367000000000001,"15min":28.050999999999998},"min":{"1min":10.449999999999999,"5min":10.449999999999999,"15min":9.0559999999999992},"max":{"1min":84.061999999999998,"5min":168.88499999999999,"15min":301.428},"last":16.616}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":24.667000000000002,"5min":28.010000000000002,"15min":28.553999999999998},"min":{"1min":12.593999999999999,"5min":8.9139999999999997,"15min":8.0549999999999997},"max":{"1min":83.186000000000007,"5min":186.81299999999999,"15min":247.31100000000001},"last":17.361999999999998},{"interface":"front","average":{"1min":19.212,"5min":20.692,"15min":24.864999999999998},"min":{"1min":8.0640000000000001,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":61.241999999999997,"5min":184.453,"15min":311.41000000000003},"last":11.105}]}]},{"osd":2,"up_from":17,"seq":73014444305,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64120,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307720,"statfs":{"total":96636764160,"available":96571105280,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":17.835999999999999,"5min":15.121,"15min":16.925000000000001},"min":{"1min":5.9960000000000004,"5min":5.9960000000000004,"15min":4.9409999999999998},"max":{"1min":111.95099999999999,"5min":149.566,"15min":339.37299999999999},"last":8.4809999999999999},{"interface":"front","average":{"1min":21.946999999999999,"5min":20.587,"15min":23.597999999999999},"min":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":4.6319999999999997},"max":{"1min":123.776,"5min":157.69300000000001,"15min":597.92399999999998},"last":11.308999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":18.486000000000001,"5min":18.428000000000001,"15min":21.236999999999998},"min":{"1min":7.4930000000000003,"5min":7.4930000000000003,"15min":4.5410000000000004},"max":{"1min":58.052999999999997,"5min":104.455,"15min":282.435},"last":12.071999999999999},{"interface":"front","average":{"1min":13.083,"5min":13.429,"15min":18.023},"min":{"1min":5.093,"5min":5.093,"15min":5.093},"max":{"1min":51.125,"5min":100.026,"15min":135.71000000000001},"last":9.7059999999999995}]},{"osd":3,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":21.981999999999999,"5min":23.928000000000001,"15min":20.510999999999999},"min":{"1min":10.015000000000001,"5min":8.1869999999999994,"15min":6.1699999999999999},"max":{"1min":47.886000000000003,"5min":160.94800000000001,"15min":160.94800000000001},"last":13.412000000000001},{"interface":"front","average":{"1min":16.581,"5min":18.431000000000001,"15min":21.231000000000002},"min":{"1min":6.5739999999999998,"5min":6.5339999999999998,"15min":6.5170000000000003},"max":{"1min":53.826000000000001,"5min":128.06100000000001,"15min":241.69},"last":9.1050000000000004}]},{"osd":4,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":20.681999999999999,"5min":18.628,"15min":19.465},"min":{"1min":7.5700000000000003,"5min":7.0599999999999996,"15min":6.5570000000000004},"max":{"1min":69.706000000000003,"5min":130.738,"15min":229.14500000000001},"last":7.6849999999999996},{"interface":"front","average":{"1min":24.361999999999998,"5min":22.751999999999999,"15min":21.934999999999999},"min":{"1min":10.942,"5min":10.942,"15min":5.9829999999999997},"max":{"1min":76.808000000000007,"5min":105.361,"15min":284.95100000000002},"last":14.02}]},{"osd":5,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":13.340999999999999,"5min":16.795999999999999,"15min":17.544},"min":{"1min":8.1159999999999997,"5min":6.0019999999999998,"15min":5.9489999999999998},"max":{"1min":43.698,"5min":144.864,"15min":247.37799999999999},"last":18.808},{"interface":"front","average":{"1min":18.547999999999998,"5min":21.937999999999999,"15min":21.414999999999999},"min":{"1min":10.209,"5min":6.7229999999999999,"15min":6.7229999999999999},"max":{"1min":57.433999999999997,"5min":143.517,"15min":218.82400000000001},"last":14.776999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":22.513000000000002,"5min":23.238,"15min":25.574000000000002},"min":{"1min":9.5549999999999997,"5min":7.7190000000000003,"15min":7.7190000000000003},"max":{"1min":64.820999999999998,"5min":133.203,"15min":288.38299999999998},"last":15.714},{"interface":"front","average":{"1min":21.914999999999999,"5min":25.452999999999999,"15min":29.690999999999999},"min":{"1min":12.843999999999999,"5min":10.220000000000001,"15min":8.25},"max":{"1min":55.670999999999999,"5min":182.745,"15min":291.435},"last":18.079000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:15 2023","interfaces":[{"interface":"back","average":{"1min":16.765999999999998,"5min":19.440000000000001,"15min":23.538},"min":{"1min":7.7519999999999998,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":45.223999999999997,"5min":162.23500000000001,"15min":302.30900000000003},"last":17.398},{"interface":"front","average":{"1min":22.626000000000001,"5min":25.323,"15min":24.372},"min":{"1min":10.784000000000001,"5min":9.3190000000000008,"15min":5.7859999999999996},"max":{"1min":66.337999999999994,"5min":136.339,"15min":279.00700000000001},"last":16.628}]}]},{"osd":1,"up_from":17,"seq":73014444305,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60356,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311484,"statfs":{"total":96636764160,"available":96574959616,"internally_reserved":0,"allocated":7139328,"data_stored":4075528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":14.016,"5min":26.524000000000001,"15min":23.024000000000001},"min":{"1min":4.9160000000000004,"5min":4.3029999999999999,"15min":4.2080000000000002},"max":{"1min":93.808999999999997,"5min":348.17200000000003,"15min":410.08199999999999},"last":10.808},{"interface":"front","average":{"1min":15.234999999999999,"5min":20.524999999999999,"15min":20.359000000000002},"min":{"1min":6.4189999999999996,"5min":5.5659999999999998,"15min":5.282},"max":{"1min":82.144999999999996,"5min":146.68700000000001,"15min":303.291},"last":17.038}]},{"osd":2,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":21.712,"15min":22.257999999999999},"min":{"1min":6.3479999999999999,"5min":4.4240000000000004,"15min":4.4240000000000004},"max":{"1min":23.940000000000001,"5min":192.226,"15min":336.255},"last":9.7899999999999991},{"interface":"front","average":{"1min":18.809999999999999,"5min":28.77,"15min":22.311},"min":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":5.2679999999999998},"max":{"1min":96.281000000000006,"5min":229.56100000000001,"15min":335.88499999999999},"last":15.611000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.492999999999999,"5min":28.306000000000001,"15min":25.484000000000002},"min":{"1min":9.2159999999999993,"5min":7.0449999999999999,"15min":6.3940000000000001},"max":{"1min":87.390000000000001,"5min":317.94999999999999,"15min":429.30399999999997},"last":14.901},{"interface":"front","average":{"1min":14.365,"5min":31.286000000000001,"15min":24.562000000000001},"min":{"1min":7.1550000000000002,"5min":6.3010000000000002,"15min":6.1589999999999998},"max":{"1min":46.652999999999999,"5min":302.00599999999997,"15min":382.79399999999998},"last":7.0599999999999996}]},{"osd":4,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":19.879999999999999,"5min":30.443999999999999,"15min":29.187000000000001},"min":{"1min":10.609,"5min":8.7759999999999998,"15min":5.9969999999999999},"max":{"1min":83.418000000000006,"5min":207.262,"15min":406.53899999999999},"last":11.369},{"interface":"front","average":{"1min":18.018999999999998,"5min":33.978999999999999,"15min":28.32},"min":{"1min":9.6039999999999992,"5min":7.4039999999999999,"15min":6.9279999999999999},"max":{"1min":96.866,"5min":417.81700000000001,"15min":417.81700000000001},"last":13.583}]},{"osd":5,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":23.068999999999999,"5min":27.93,"15min":28.184999999999999},"min":{"1min":11.724,"5min":10.911,"15min":7.3049999999999997},"max":{"1min":106.518,"5min":128.53899999999999,"15min":390.11000000000001},"last":12.308999999999999},{"interface":"front","average":{"1min":21.890000000000001,"5min":33.082000000000001,"15min":27.628},"min":{"1min":10.880000000000001,"5min":8.8249999999999993,"15min":7.0350000000000001},"max":{"1min":102.294,"5min":323.339,"15min":357.44799999999998},"last":16.295999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":22.527000000000001,"5min":29.898,"15min":31.405999999999999},"min":{"1min":11.798,"5min":10.257,"15min":7.9279999999999999},"max":{"1min":90.643000000000001,"5min":188.14400000000001,"15min":420.565},"last":12.965999999999999},{"interface":"front","average":{"1min":26.419,"5min":35.195999999999998,"15min":32.094999999999999},"min":{"1min":14.733000000000001,"5min":12.470000000000001,"15min":8.8680000000000003},"max":{"1min":94.477999999999994,"5min":264.226,"15min":340.71100000000001},"last":18.427}]},{"osd":7,"last update":"Thu Dec 7 07:55:11 2023","interfaces":[{"interface":"back","average":{"1min":17.436,"5min":24.896999999999998,"15min":27.373999999999999},"min":{"1min":8.2690000000000001,"5min":6.6500000000000004,"15min":6.6500000000000004},"max":{"1min":77.435000000000002,"5min":200.01900000000001,"15min":442.14100000000002},"last":8.9670000000000005},{"interface":"front","average":{"1min":24.981000000000002,"5min":32.478999999999999,"15min":30.864000000000001},"min":{"1min":13.936999999999999,"5min":10.494999999999999,"15min":8.8699999999999992},"max":{"1min":98.965999999999994,"5min":267.07999999999998,"15min":533.87199999999996},"last":19.038}]}]},{"osd":0,"up_from":16,"seq":68719477009,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59964,"kb_used_data":6580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311876,"statfs":{"total":96636764160,"available":96575361024,"internally_reserved":0,"allocated":6737920,"data_stored":3677688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.449,"5min":20.334,"15min":21.199999999999999},"min":{"1min":7.2389999999999999,"5min":5.3540000000000001,"15min":5.3529999999999998},"max":{"1min":52.637,"5min":230.792,"15min":230.792},"last":14.691000000000001},{"interface":"front","average":{"1min":17.687999999999999,"5min":22.419,"15min":27.146999999999998},"min":{"1min":8.0589999999999993,"5min":6.9210000000000003,"15min":6.9210000000000003},"max":{"1min":68.528000000000006,"5min":119.28,"15min":289.82400000000001},"last":16.363}]},{"osd":2,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":15.130000000000001,"5min":24.141999999999999,"15min":23.670999999999999},"min":{"1min":9.3160000000000007,"5min":8.8759999999999994,"15min":7.9080000000000004},"max":{"1min":26.367000000000001,"5min":201.53800000000001,"15min":201.53800000000001},"last":15.715999999999999},{"interface":"front","average":{"1min":9.2829999999999995,"5min":17.088999999999999,"15min":16.57},"min":{"1min":5.8230000000000004,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":29.462,"5min":202.648,"15min":214.94},"last":7.9000000000000004}]},{"osd":3,"last update":"Thu Dec 7 07:55:13 2023","interfaces":[{"interface":"back","average":{"1min":20.521000000000001,"5min":22.611000000000001,"15min":27.707000000000001},"min":{"1min":6.1550000000000002,"5min":5.3440000000000003,"15min":5.0899999999999999},"max":{"1min":240.876,"5min":283.43000000000001,"15min":932.75999999999999},"last":10.137},{"interface":"front","average":{"1min":20.475000000000001,"5min":26.277999999999999,"15min":27.437000000000001},"min":{"1min":11.481,"5min":8.0809999999999995,"15min":8.0809999999999995},"max":{"1min":78.986999999999995,"5min":256.267,"15min":300.13999999999999},"last":17.673999999999999}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":23.826000000000001,"5min":21.247,"15min":28.013000000000002},"min":{"1min":7.1529999999999996,"5min":6.7930000000000001,"15min":6.4649999999999999},"max":{"1min":153.63200000000001,"5min":153.63200000000001,"15min":1049.731},"last":12.077999999999999},{"interface":"front","average":{"1min":27.728000000000002,"5min":27.978999999999999,"15min":28.177},"min":{"1min":13.824,"5min":12.833,"15min":9.5399999999999991},"max":{"1min":108.58799999999999,"5min":158.40100000000001,"15min":283.43700000000001},"last":18.302}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":24.460999999999999,"5min":30.645,"15min":28.616},"min":{"1min":10.550000000000001,"5min":10.550000000000001,"15min":5.8300000000000001},"max":{"1min":105.548,"5min":248.34299999999999,"15min":255.328},"last":19.510999999999999},{"interface":"front","average":{"1min":20.219999999999999,"5min":23.016999999999999,"15min":30.934000000000001},"min":{"1min":6.8079999999999998,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":185.125,"5min":264.34899999999999,"15min":1217.117},"last":9.0890000000000004}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.291,"5min":21.850000000000001,"15min":31.241},"min":{"1min":8.1869999999999994,"5min":7.8789999999999996,"15min":7.0350000000000001},"max":{"1min":28.216999999999999,"5min":154.68199999999999,"15min":1252.499},"last":12.98},{"interface":"front","average":{"1min":20.312000000000001,"5min":30.834,"15min":31.664999999999999},"min":{"1min":14.042,"5min":13.167999999999999,"15min":11.638},"max":{"1min":33.158000000000001,"5min":263.43200000000002,"15min":263.43200000000002},"last":20.530000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":22.698,"5min":32.343000000000004,"15min":32.625999999999998},"min":{"1min":15.154999999999999,"5min":13.768000000000001,"15min":13.212},"max":{"1min":37.085000000000001,"5min":140.50200000000001,"15min":279.38900000000001},"last":21.283999999999999},{"interface":"front","average":{"1min":14.250999999999999,"5min":24.257999999999999,"15min":23.280999999999999},"min":{"1min":7.8200000000000003,"5min":7.1459999999999999,"15min":6.2990000000000004},"max":{"1min":27.417000000000002,"5min":262.07299999999998,"15min":262.07299999999998},"last":11.355}]}]},{"osd":6,"up_from":14,"seq":60129542417,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59960,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311880,"statfs":{"total":96636764160,"available":96575365120,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":11.407,"5min":16.442,"15min":18.248999999999999},"min":{"1min":5.4500000000000002,"5min":5.4500000000000002,"15min":4.8579999999999997},"max":{"1min":30.831,"5min":227.49000000000001,"15min":227.49000000000001},"last":9.1170000000000009},{"interface":"front","average":{"1min":19.172999999999998,"5min":22.824000000000002,"15min":24.23},"min":{"1min":6.343,"5min":6.343,"15min":5.9909999999999997},"max":{"1min":80.697000000000003,"5min":228.13399999999999,"15min":229.405},"last":13.260999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":14.680999999999999,"5min":21.905999999999999,"15min":22.215},"min":{"1min":8.3089999999999993,"5min":6.2599999999999998,"15min":5.9390000000000001},"max":{"1min":24.978000000000002,"5min":228.745,"15min":234.59399999999999},"last":10.465},{"interface":"front","average":{"1min":10.353,"5min":18.015999999999998,"15min":17.309000000000001},"min":{"1min":5.702,"5min":4.8090000000000002,"15min":4.6100000000000003},"max":{"1min":18.276,"5min":220.78800000000001,"15min":220.78800000000001},"last":7.9290000000000003}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":11.086,"5min":19.494,"15min":18.353000000000002},"min":{"1min":7.4279999999999999,"5min":5.8680000000000003,"15min":4.4210000000000003},"max":{"1min":30.722999999999999,"5min":156.13499999999999,"15min":215.05699999999999},"last":7.2729999999999997},{"interface":"front","average":{"1min":16.837,"5min":26.483000000000001,"15min":25.437999999999999},"min":{"1min":10.422000000000001,"5min":7.9950000000000001,"15min":7.9950000000000001},"max":{"1min":26.327000000000002,"5min":232.45400000000001,"15min":232.45400000000001},"last":18.245999999999999}]},{"osd":3,"last update":"Thu Dec 7 07:55:27 2023","interfaces":[{"interface":"back","average":{"1min":12.009,"5min":25.157,"15min":22.254999999999999},"min":{"1min":6.7590000000000003,"5min":6.024,"15min":6.024},"max":{"1min":26.135999999999999,"5min":248.97900000000001,"15min":248.97900000000001},"last":13.875},{"interface":"front","average":{"1min":14.058,"5min":25.068000000000001,"15min":22.209},"min":{"1min":9.5719999999999992,"5min":6.0860000000000003,"15min":6.0860000000000003},"max":{"1min":27.628,"5min":236.92699999999999,"15min":236.92699999999999},"last":16.847999999999999}]},{"osd":4,"last update":"Thu Dec 7 07:56:00 2023","interfaces":[{"interface":"back","average":{"1min":35.658000000000001,"5min":30.891999999999999,"15min":24.102},"min":{"1min":8.6880000000000006,"5min":6.3899999999999997,"15min":6.3090000000000002},"max":{"1min":246.28299999999999,"5min":276.92500000000001,"15min":276.92500000000001},"last":15.407},{"interface":"front","average":{"1min":17.623999999999999,"5min":27.646000000000001,"15min":23.425000000000001},"min":{"1min":6.7910000000000004,"5min":6.7910000000000004,"15min":6.7910000000000004},"max":{"1min":47.604999999999997,"5min":179.64099999999999,"15min":220.881},"last":17.459}]},{"osd":5,"last update":"Thu Dec 7 07:56:00 2023","interfaces":[{"interface":"back","average":{"1min":30.048999999999999,"5min":26.992000000000001,"15min":25.532},"min":{"1min":7.9720000000000004,"5min":7.8570000000000002,"15min":7.8570000000000002},"max":{"1min":152.00399999999999,"5min":237.53700000000001,"15min":237.53700000000001},"last":21.236999999999998},{"interface":"front","average":{"1min":28.152999999999999,"5min":28.809000000000001,"15min":25.042999999999999},"min":{"1min":8.6590000000000007,"5min":8.6590000000000007,"15min":8.0099999999999998},"max":{"1min":127.7,"5min":239.673,"15min":239.673},"last":11.782999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":26.934999999999999,"5min":30.998999999999999,"15min":25.236999999999998},"min":{"1min":10.981999999999999,"5min":8.4890000000000008,"15min":7.3090000000000002},"max":{"1min":107.471,"5min":236.26499999999999,"15min":236.26499999999999},"last":22.013999999999999},{"interface":"front","average":{"1min":28.969999999999999,"5min":31.443999999999999,"15min":28.527999999999999},"min":{"1min":17.210999999999999,"5min":10.239000000000001,"15min":10.239000000000001},"max":{"1min":102.93600000000001,"5min":253.29300000000001,"15min":253.29300000000001},"last":16.036999999999999}]}]},{"osd":7,"up_from":13,"seq":55834575123,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":10.605,"5min":11.368,"15min":15.794},"min":{"1min":5.6929999999999996,"5min":4.3899999999999997,"15min":4.29},"max":{"1min":17.640999999999998,"5min":127.59,"15min":214.32599999999999},"last":13.346},{"interface":"front","average":{"1min":10.164,"5min":10.188000000000001,"15min":15.063000000000001},"min":{"1min":4.3659999999999997,"5min":4.3659999999999997,"15min":4.3659999999999997},"max":{"1min":20.850000000000001,"5min":47.606000000000002,"15min":199.94499999999999},"last":4.5179999999999998}]},{"osd":1,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":12.381,"5min":16.414000000000001,"15min":19.100000000000001},"min":{"1min":6.3719999999999999,"5min":5.8289999999999997,"15min":5.3849999999999998},"max":{"1min":25.638999999999999,"5min":157.16,"15min":198.512},"last":9.4100000000000001},{"interface":"front","average":{"1min":12.368,"5min":17.263999999999999,"15min":19.622},"min":{"1min":7.4349999999999996,"5min":5.7270000000000003,"15min":5.6929999999999996},"max":{"1min":18.352,"5min":167.78200000000001,"15min":167.78200000000001},"last":15.122}]},{"osd":2,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":13.574,"5min":18.367999999999999,"15min":18.265000000000001},"min":{"1min":7.3330000000000002,"5min":6.0049999999999999,"15min":5.9669999999999996},"max":{"1min":23.154,"5min":282.12099999999998,"15min":282.12099999999998},"last":7.9359999999999999},{"interface":"front","average":{"1min":13.243,"5min":17.161000000000001,"15min":19.664000000000001},"min":{"1min":6.9939999999999998,"5min":6.9939999999999998,"15min":6.9939999999999998},"max":{"1min":20.405000000000001,"5min":154.376,"15min":216.28299999999999},"last":13.999000000000001}]},{"osd":3,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":14.811999999999999,"5min":16.736999999999998,"15min":20.478999999999999},"min":{"1min":9.3559999999999999,"5min":6.4939999999999998,"15min":6.3410000000000002},"max":{"1min":21.123999999999999,"5min":84.263000000000005,"15min":159.934},"last":17.777999999999999},{"interface":"front","average":{"1min":15.291,"5min":18.553999999999998,"15min":20.609000000000002},"min":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":6.9409999999999998},"max":{"1min":28.382999999999999,"5min":147.85400000000001,"15min":157.43100000000001},"last":10.199999999999999}]},{"osd":4,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":16.672000000000001,"5min":17.713999999999999,"15min":21.768000000000001},"min":{"1min":9.3829999999999991,"5min":7.2110000000000003,"15min":6.7000000000000002},"max":{"1min":33.926000000000002,"5min":94.227000000000004,"15min":204.71199999999999},"last":10.981},{"interface":"front","average":{"1min":18.524999999999999,"5min":18.350000000000001,"15min":21.297999999999998},"min":{"1min":9.1869999999999994,"5min":7.7839999999999998,"15min":7.1269999999999998},"max":{"1min":52.731000000000002,"5min":93.356999999999999,"15min":259.23200000000003},"last":11.6}]},{"osd":5,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.744,"5min":21.847000000000001,"15min":25.094000000000001},"min":{"1min":10.131,"5min":8.9830000000000005,"15min":8.9830000000000005},"max":{"1min":92.340999999999994,"5min":189.49000000000001,"15min":239.32599999999999},"last":15.801},{"interface":"front","average":{"1min":15.037000000000001,"5min":18.690999999999999,"15min":23.567},"min":{"1min":8.3529999999999998,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":29.849,"5min":157.60400000000001,"15min":224.91900000000001},"last":19.064}]},{"osd":6,"last update":"Thu Dec 7 07:55:18 2023","interfaces":[{"interface":"back","average":{"1min":19.780999999999999,"5min":20.637,"15min":24.367999999999999},"min":{"1min":12.939,"5min":9.9390000000000001,"15min":8.2750000000000004},"max":{"1min":49.506999999999998,"5min":66.572999999999993,"15min":192.76300000000001},"last":16.414999999999999},{"interface":"front","average":{"1min":20.835000000000001,"5min":23.234999999999999,"15min":29.07},"min":{"1min":13.638999999999999,"5min":13.432,"15min":10.176},"max":{"1min":34.186999999999998,"5min":92.688999999999993,"15min":216.917},"last":18.452000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575122,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82560,"kb_used_data":6968,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289280,"statfs":{"total":96636764160,"available":96552222720,"internally_reserved":0,"allocated":7135232,"data_stored":4075523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":11.474,"5min":12.582000000000001,"15min":22.064},"min":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":4.2409999999999997},"max":{"1min":27.518000000000001,"5min":130.309,"15min":1086.095},"last":6.9710000000000001},{"interface":"front","average":{"1min":11.550000000000001,"5min":16.629999999999999,"15min":20.43},"min":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"max":{"1min":38.128999999999998,"5min":141.636,"15min":403.21899999999999},"last":5.3300000000000001}]},{"osd":1,"last update":"Thu Dec 7 07:55:36 2023","interfaces":[{"interface":"back","average":{"1min":20.469000000000001,"5min":15.369,"15min":17.274999999999999},"min":{"1min":6.234,"5min":5.4050000000000002,"15min":4.819},"max":{"1min":130.52500000000001,"5min":130.52500000000001,"15min":310.18299999999999},"last":12.614000000000001},{"interface":"front","average":{"1min":23.145,"5min":17.341000000000001,"15min":20.655000000000001},"min":{"1min":7.8479999999999999,"5min":6.4640000000000004,"15min":4.9290000000000003},"max":{"1min":113.23099999999999,"5min":113.23099999999999,"15min":263.452},"last":8.0429999999999993}]},{"osd":2,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.693000000000001,"5min":19.597000000000001,"15min":21.677},"min":{"1min":6.149,"5min":6.149,"15min":6.149},"max":{"1min":70.923000000000002,"5min":142.77699999999999,"15min":250.30600000000001},"last":9.8870000000000005},{"interface":"front","average":{"1min":16.704999999999998,"5min":18.449000000000002,"15min":20.291},"min":{"1min":8.2810000000000006,"5min":7.8360000000000003,"15min":6.4059999999999997},"max":{"1min":60.683,"5min":148.40100000000001,"15min":300.988},"last":13.340999999999999}]},{"osd":3,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":17.314,"5min":22.145,"15min":23.927},"min":{"1min":7.9210000000000003,"5min":7.4370000000000003,"15min":6.9809999999999999},"max":{"1min":55.113999999999997,"5min":143.53899999999999,"15min":305.74900000000002},"last":10.494999999999999},{"interface":"front","average":{"1min":20.635999999999999,"5min":24.911999999999999,"15min":26.052},"min":{"1min":10.500999999999999,"5min":10.500999999999999,"15min":10.500999999999999},"max":{"1min":71.872,"5min":144.41800000000001,"15min":326.85899999999998},"last":14.797000000000001}]},{"osd":4,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":15.747999999999999,"5min":21.678000000000001,"15min":24.283000000000001},"min":{"1min":7.5999999999999996,"5min":6.5899999999999999,"15min":6.5899999999999999},"max":{"1min":68.769999999999996,"5min":136.691,"15min":320.23099999999999},"last":17.859000000000002},{"interface":"front","average":{"1min":17.957000000000001,"5min":24.734000000000002,"15min":23.757999999999999},"min":{"1min":10.146000000000001,"5min":8.7840000000000007,"15min":6.6929999999999996},"max":{"1min":39.585999999999999,"5min":167.351,"15min":313.42500000000001},"last":11.989000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":16.943999999999999,"5min":25.558,"15min":30.716999999999999},"min":{"1min":10.445,"5min":9.1029999999999998,"15min":8.9280000000000008},"max":{"1min":35.082999999999998,"5min":201.87299999999999,"15min":320.84199999999998},"last":18.489999999999998},{"interface":"front","average":{"1min":18.722000000000001,"5min":25.786999999999999,"15min":30.187999999999999},"min":{"1min":11.132,"5min":8.3030000000000008,"15min":6.3630000000000004},"max":{"1min":32.395000000000003,"5min":192.81800000000001,"15min":365.91899999999998},"last":16.873000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:55:17 2023","interfaces":[{"interface":"back","average":{"1min":13.975,"5min":19.853000000000002,"15min":21.242999999999999},"min":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":6.484},"max":{"1min":24.908999999999999,"5min":203.636,"15min":286.029},"last":11.265000000000001},{"interface":"front","average":{"1min":17.344999999999999,"5min":24.433,"15min":28.132999999999999},"min":{"1min":8.4990000000000006,"5min":8.4990000000000006,"15min":8.4990000000000006},"max":{"1min":31.541,"5min":204.45500000000001,"15min":300.18200000000002},"last":14.012}]}]},{"osd":4,"up_from":13,"seq":55834575121,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59960,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311880,"statfs":{"total":96636764160,"available":96575365120,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":10.369999999999999,"5min":13.005000000000001,"15min":17.228999999999999},"min":{"1min":4.5499999999999998,"5min":4.5499999999999998,"15min":4.3230000000000004},"max":{"1min":22.457999999999998,"5min":98.765000000000001,"15min":301.959},"last":9.8059999999999992},{"interface":"front","average":{"1min":11.028,"5min":15.567,"15min":21.140999999999998},"min":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"max":{"1min":24.654,"5min":110.508,"15min":363.75099999999998},"last":6.7389999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":14.859999999999999,"5min":15.579000000000001,"15min":20.225000000000001},"min":{"1min":6.4900000000000002,"5min":6.4900000000000002,"15min":5.577},"max":{"1min":57.762,"5min":121.346,"15min":219.19300000000001},"last":10.487},{"interface":"front","average":{"1min":13.955,"5min":16.913,"15min":20.986000000000001},"min":{"1min":6.3449999999999998,"5min":6.3449999999999998,"15min":6.3209999999999997},"max":{"1min":38.786000000000001,"5min":111.107,"15min":183.60400000000001},"last":11.997999999999999}]},{"osd":2,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.715999999999999,"5min":21.856000000000002,"15min":20.428999999999998},"min":{"1min":8.4719999999999995,"5min":7.8289999999999997,"15min":6.8710000000000004},"max":{"1min":20.859000000000002,"5min":176.452,"15min":176.452},"last":8.9740000000000002},{"interface":"front","average":{"1min":15.119999999999999,"5min":22.337,"15min":21.960000000000001},"min":{"1min":8.2330000000000005,"5min":7.6020000000000003,"15min":7.5940000000000003},"max":{"1min":19.547000000000001,"5min":239.935,"15min":239.935},"last":16.010000000000002}]},{"osd":3,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":9.5579999999999998,"5min":18.57,"15min":22.199000000000002},"min":{"1min":5.968,"5min":5.968,"15min":5.968},"max":{"1min":13.237,"5min":177.48699999999999,"15min":246.523},"last":8.0239999999999991},{"interface":"front","average":{"1min":14.81,"5min":20.649000000000001,"15min":18.181999999999999},"min":{"1min":7.4720000000000004,"5min":6.274,"15min":5.4240000000000004},"max":{"1min":20.555,"5min":183.75399999999999,"15min":183.75399999999999},"last":13.4}]},{"osd":5,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":12.641,"5min":15.577,"15min":19.568999999999999},"min":{"1min":8.4770000000000003,"5min":6.327,"15min":6.327},"max":{"1min":20.013000000000002,"5min":123.998,"15min":148.21000000000001},"last":14.137},{"interface":"front","average":{"1min":18.177,"5min":22.536000000000001,"15min":23.007999999999999},"min":{"1min":13.364000000000001,"5min":9.1270000000000007,"15min":6.46},"max":{"1min":25.591000000000001,"5min":224.55500000000001,"15min":224.55500000000001},"last":16.669}]},{"osd":6,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":13.776999999999999,"5min":20.428000000000001,"15min":23.344000000000001},"min":{"1min":8.4740000000000002,"5min":8.2129999999999992,"15min":7.8079999999999998},"max":{"1min":22.021000000000001,"5min":158.36099999999999,"15min":268.392},"last":11.41},{"interface":"front","average":{"1min":17.134,"5min":25.366,"15min":26.675999999999998},"min":{"1min":13.542999999999999,"5min":10.300000000000001,"15min":9.9299999999999997},"max":{"1min":23.768000000000001,"5min":243.40700000000001,"15min":243.40700000000001},"last":14.702999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:55:16 2023","interfaces":[{"interface":"back","average":{"1min":17.846,"5min":24.731999999999999,"15min":26.576000000000001},"min":{"1min":10.978,"5min":9.9939999999999998,"15min":9.2319999999999993},"max":{"1min":25.713000000000001,"5min":225.375,"15min":261.75099999999998},"last":15.284000000000001},{"interface":"front","average":{"1min":13.435,"5min":19.010999999999999,"15min":19.09},"min":{"1min":9.1319999999999997,"5min":7.181,"15min":6.7110000000000003},"max":{"1min":20.039000000000001,"5min":167.53700000000001,"15min":178.39699999999999},"last":17.366}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:56:17.854 INFO:tasks.ceph:pgid 11.14 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:56:17.854 INFO:tasks.ceph:pgid 11.1e last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:56:17.854 INFO:tasks.ceph:pgid 11.6 last_scrub_stamp 2023-12-07T07:34:40.816370+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=34, tm_sec=40, tm_wday=3, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=7, tm_hour=7, tm_min=55, tm_sec=48, tm_wday=3, tm_yday=341, tm_isdst=0) 2023-12-07T07:56:17.855 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-07T07:56:21.174 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:47.623 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.176 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:47.625 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.190 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:23:47.639 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.221 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:47.659 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.222 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:47.671 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.222 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:47.660 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.226 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:47.675 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.233 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:47.671 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:21.235 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:23:47.685 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.215 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:02.663 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.218 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:02.667 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.234 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:02.683 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.267 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:02.716 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.270 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:02.708 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.271 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:02.720 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.271 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:02.709 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.284 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:02.722 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:36.287 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:02.736 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:37.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T07:56:37.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.953+0000 7fd1335a9640 10 monclient: get_monmap_and_config 2023-12-07T07:56:37.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.954+0000 7fd1335a9640 10 monclient: build_initial_monmap 2023-12-07T07:56:37.956 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.954+0000 7fd1335a9640 10 monclient: monmap: 2023-12-07T07:56:37.956 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:56:37.956 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:56:37.955078+0000 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:56:37.955078+0000 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:56:37.957 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:37.958 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.954+0000 7fd1335a9640 1 Processor -- start 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.954+0000 7fd1335a9640 1 -- start start 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.954+0000 7fd1335a9640 10 monclient: init 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient: _reopen_session rank -1 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd12c0af0f0 addr v1:172.21.15.154:6789/0 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd12c0af4d0 addr v1:172.21.15.97:6789/0 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fd12c072760 0x7fd12c14a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient(hunting): picked mon.noname-a con 0x7fd12c072760 addr v2:172.21.15.97:3300/0 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c070b60 con 0x7fd12c0af4d0 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c072340 con 0x7fd12c0af0f0 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12c0afe80 con 0x7fd12c072760 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient(hunting): _renew_subs 2023-12-07T07:56:37.959 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.955+0000 7fd1335a9640 10 monclient(hunting): authenticate will time out at 2720.333780s 2023-12-07T07:56:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.956+0000 7fd131345640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd12c0af0f0 0x7fd12c0ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.97:35450/0 (socket says 172.21.15.97:35450) 2023-12-07T07:56:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.956+0000 7fd131345640 1 -- 172.21.15.97:0/1208492145 learned_addr learned my addr 172.21.15.97:0/1208492145 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-07T07:56:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.956+0000 7fd131b46640 1 --2- 172.21.15.97:0/1208492145 >> v2:172.21.15.97:3300/0 conn(0x7fd12c072760 0x7fd12c14a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.957+0000 7fd131b46640 10 monclient(hunting): get_auth_request con 0x7fd12c072760 auth_method 0 2023-12-07T07:56:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.957+0000 7fd131b46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.957+0000 7fd131b46640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:37.960 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.957+0000 7fd131b46640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.958+0000 7fd123fff640 1 -- 172.21.15.97:0/1208492145 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691289547 0 0) 0x7fd12c072340 con 0x7fd12c0af0f0 2023-12-07T07:56:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.958+0000 7fd123fff640 10 monclient(hunting): _negotiate 2023-12-07T07:56:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.958+0000 7fd123fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.958+0000 7fd123fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.958+0000 7fd123fff640 10 monclient(hunting): my global_id is 20275 2023-12-07T07:56:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.958+0000 7fd123fff640 1 -- 172.21.15.97:0/1208492145 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd118003520 con 0x7fd12c0af0f0 2023-12-07T07:56:37.961 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.959+0000 7fd131b46640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:37.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.959+0000 7fd131b46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:37.962 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.959+0000 7fd131b46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 10 monclient(hunting): handle_auth_done global_id 16227 payload 274 2023-12-07T07:56:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 1 -- 172.21.15.97:0/1208492145 >> v1:172.21.15.154:6789/0 conn(0x7fd12c0af0f0 legacy=0x7fd12c0ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 1 -- 172.21.15.97:0/1208492145 >> v1:172.21.15.97:6789/0 conn(0x7fd12c0af4d0 legacy=0x7fd12c142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 10 monclient: _finish_hunting 0 2023-12-07T07:56:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 1 monclient: found mon.noname-a 2023-12-07T07:56:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:56:37.963 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:56:37.964 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.961+0000 7fd131b46640 1 -- 172.21.15.97:0/1208492145 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c13c600 con 0x7fd12c072760 2023-12-07T07:56:37.966 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.964+0000 7fd131b46640 1 --2- 172.21.15.97:0/1208492145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c072760 0x7fd12c14a3e0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fd124009d90 tx=0x7fd12402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152ca1f3f5c703e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 1 -- 172.21.15.97:0/1208492145 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12402cbd0 con 0x7fd12c072760 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: dump: 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:37.970 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: _finish_auth 0 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: _check_auth_tickets 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 1 -- 172.21.15.97:0/1208492145 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12402cd30 con 0x7fd12c072760 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd1335a9640 5 monclient: authenticate success, global_id 16227 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd1335a9640 10 monclient: get_monmap_and_config success 2023-12-07T07:56:37.971 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 1 -- 172.21.15.97:0/1208492145 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12402cea0 con 0x7fd12c072760 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd1335a9640 4 set_mon_vals no callback set 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.967+0000 7fd123fff640 10 monclient: dump: 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:37.972 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 10 monclient: shutdown 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 1 -- 172.21.15.97:0/1208492145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c072760 msgr2=0x7fd12c14a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1208492145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c072760 0x7fd12c14a3e0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fd124009d90 tx=0x7fd12402be20 comp rx=0 tx=0).stop 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 1 -- 172.21.15.97:0/1208492145 shutdown_connections 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1208492145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c072760 0x7fd12c14a3e0 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 1 -- 172.21.15.97:0/1208492145 >> 172.21.15.97:0/1208492145 conn(0x7fd12c0b01e0 msgr2=0x7fd12c0b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 1 -- 172.21.15.97:0/1208492145 shutdown_connections 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.968+0000 7fd1335a9640 1 -- 172.21.15.97:0/1208492145 wait complete. 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 10 monclient: build_initial_monmap 2023-12-07T07:56:37.976 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 10 monclient: monmap: 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 1 librados: starting msgr at 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 1 librados: starting objecter 2023-12-07T07:56:37.977 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 1 Processor -- start 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 1 -- start start 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 1 librados: setting wanted keys 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 1 librados: calling monclient init 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.969+0000 7fd1335a9640 10 monclient: init 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient: _reopen_session rank -1 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd12c072760 0x7fd12c1d3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): picked mon.noname-b con 0x7fd12c072760 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd12c0af0f0 0x7fd12c139080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): picked mon.noname-c con 0x7fd12c0af0f0 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c13f610 0x7fd12c1395c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): picked mon.noname-a con 0x7fd12c13f610 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1d2c80 con 0x7fd12c13f610 2023-12-07T07:56:37.978 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1d36c0 con 0x7fd12c072760 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1d28a0 con 0x7fd12c0af0f0 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): _renew_subs 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd1335a9640 10 monclient(hunting): authenticate will time out at 2720.348692s 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd131b46640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c13f610 0x7fd12c1395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.970+0000 7fd130b44640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd12c0af0f0 0x7fd12c139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd12c072760 0x7fd12c1d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c13f610 0x7fd12c1395c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45896/0 (socket says 172.21.15.97:45896) 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 1 -- 172.21.15.97:0/1701760359 learned_addr learned my addr 172.21.15.97:0/1701760359 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 10 monclient(hunting): get_auth_request con 0x7fd12c13f610 auth_method 0 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:37.979 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 10 monclient(hunting): get_auth_request con 0x7fd12c072760 auth_method 0 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd130b44640 10 monclient(hunting): get_auth_request con 0x7fd12c0af0f0 auth_method 0 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd130b44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd130b44640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd130b44640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131b46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.971+0000 7fd131345640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.972+0000 7fd130b44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.972+0000 7fd130b44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:37.980 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.972+0000 7fd130b44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:37.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 10 monclient(hunting): handle_auth_done global_id 16233 payload 824 2023-12-07T07:56:37.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 1 -- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd12c0af0f0 msgr2=0x7fd12c139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:37.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd12c0af0f0 0x7fd12c139080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:37.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 1 -- 172.21.15.97:0/1701760359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd12c072760 msgr2=0x7fd12c1d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:37.981 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd12c072760 0x7fd12c1d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 10 monclient: _finish_hunting 0 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 1 monclient: found mon.noname-a 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.97:3300/0 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.974+0000 7fd131b46640 1 -- 172.21.15.97:0/1701760359 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124009a70 con 0x7fd12c13f610 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.977+0000 7fd131b46640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c13f610 0x7fd12c1395c0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7fd124009eb0 tx=0x7fd124004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.977+0000 7fd121ffb640 10 client.?.objecter ms_handle_connect 0x7fd12c13f610 2023-12-07T07:56:37.982 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.977+0000 7fd121ffb640 10 client.?.objecter resend_mon_ops 2023-12-07T07:56:37.998 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 1 -- 172.21.15.97:0/1701760359 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12402cea0 con 0x7fd12c13f610 2023-12-07T07:56:37.999 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:37.999 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-07T07:56:37.999 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: dump: 2023-12-07T07:56:37.999 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:37.999 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:37.999 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:37.999 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: _finish_auth 0 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: _check_auth_tickets 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 1 -- 172.21.15.97:0/1701760359 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12403e610 con 0x7fd12c13f610 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 5 monclient: authenticate success, global_id 16233 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 10 monclient: _renew_subs 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:56:38.000 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c1d5da0 con 0x7fd12c13f610 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 1 -- 172.21.15.97:0/1701760359 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12403e9c0 con 0x7fd12c13f610 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd121ffb640 10 monclient: dump: 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.001 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1237fe640 4 set_mon_vals no callback set 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 10 client.16233.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 10 monclient: _renew_subs 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c1d6250 con 0x7fd12c13f610 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.995+0000 7fd1335a9640 1 librados: init done 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.996+0000 7fd1335a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.996+0000 7fd1335a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.996+0000 7fd1335a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.97:3300/0 2023-12-07T07:56:38.002 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:37.996+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c1d2ac0 con 0x7fd12c13f610 2023-12-07T07:56:38.234 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.232+0000 7fd121ffb640 1 -- 172.21.15.97:0/1701760359 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd12403b050 con 0x7fd12c13f610 2023-12-07T07:56:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.232+0000 7fd121ffb640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd11006c820 0x7fd11006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.232+0000 7fd131345640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd11006c820 0x7fd11006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.232+0000 7fd131345640 10 monclient: get_auth_request con 0x7fd11006c820 auth_method 0 2023-12-07T07:56:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.233+0000 7fd131345640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd11006c820 0x7fd11006ece0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd1280011e0 tx=0x7fd128007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:38.235 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.233+0000 7fd121ffb640 10 client.16233.objecter ms_handle_connect 0x7fd11006c820 2023-12-07T07:56:38.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.244+0000 7fd121ffb640 1 -- 172.21.15.97:0/1701760359 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fd12407afc0 con 0x7fd12c13f610 2023-12-07T07:56:38.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.244+0000 7fd121ffb640 10 client.16233.objecter ms_dispatch 0x7fd12c000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:56:38.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.244+0000 7fd121ffb640 3 client.16233.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:56:38.247 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.244+0000 7fd121ffb640 3 client.16233.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:56:38.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.245+0000 7fd121ffb640 20 client.16233.objecter dump_active .. 0 homeless 2023-12-07T07:56:38.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.245+0000 7fd121ffb640 1 -- 172.21.15.97:0/1701760359 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd124077300 con 0x7fd12c13f610 2023-12-07T07:56:38.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.245+0000 7fd121ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:38.248 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.245+0000 7fd121ffb640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:56:38.413 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.410+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 --> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd12c070fe0 con 0x7fd11006c820 2023-12-07T07:56:38.418 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.415+0000 7fd121ffb640 1 -- 172.21.15.97:0/1701760359 <== mgr.4102 v2:172.21.15.97:6800/3287182838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152049 (secure 0 0 0) 0x7fd1280014d0 con 0x7fd11006c820 2023-12-07T07:56:38.419 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-07T07:56:38.420 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.417+0000 7fd1335a9640 10 librados: watch_flush enter 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 10 librados: watch_flush exit 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 20 client.16233.objecter shutdown clearing up homeless session... 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 10 client.16233.objecter successfully canceled tick 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd11006c820 msgr2=0x7fd11006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd11006c820 0x7fd11006ece0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd1280011e0 tx=0x7fd128007d30 comp rx=0 tx=0).stop 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 10 monclient: shutdown 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c13f610 msgr2=0x7fd12c1395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c13f610 0x7fd12c1395c0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7fd124009eb0 tx=0x7fd124004a40 comp rx=0 tx=0).stop 2023-12-07T07:56:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 shutdown_connections 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fd11006c820 0x7fd11006ece0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd12c13f610 0x7fd12c1395c0 unknown :-1 s=CLOSED pgs=3247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fd12c0af0f0 0x7fd12c139080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 --2- 172.21.15.97:0/1701760359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd12c072760 0x7fd12c1d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 >> 172.21.15.97:0/1701760359 conn(0x7fd12c0b01e0 msgr2=0x7fd12c0b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 shutdown_connections 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 -- 172.21.15.97:0/1701760359 wait complete. 2023-12-07T07:56:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.418+0000 7fd1335a9640 1 librados: shutdown 2023-12-07T07:56:38.432 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":1616,"stamp":"2023-12-07T07:56:37.974345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542936,"kb_used_data":53400,"kb_used_omap":12,"kb_used_meta":489459,"kb_avail":754431784,"statfs":{"total":773094113280,"available":772538146816,"internally_reserved":0,"allocated":54681600,"data_stored":30217154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":501206576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002565"},"pg_stats":[{"pgid":"11.17","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:05.577005+0000","last_change":"2023-12-07T07:56:05.577005+0000","last_active":"2023-12-07T07:56:05.577005+0000","last_peered":"2023-12-07T07:56:05.577005+0000","last_clean":"2023-12-07T07:56:05.577005+0000","last_became_active":"2023-12-07T07:34:49.221580+0000","last_became_peered":"2023-12-07T07:34:49.221580+0000","last_unstale":"2023-12-07T07:56:05.577005+0000","last_undegraded":"2023-12-07T07:56:05.577005+0000","last_fullsized":"2023-12-07T07:56:05.577005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:05.575074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:05.575074+0000","last_clean_scrub_stamp":"2023-12-07T07:56:05.575074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:51:34.125625+0000","scrub_duration":0.070435624000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":875,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:07.470832+0000","last_change":"2023-12-07T07:56:07.470832+0000","last_active":"2023-12-07T07:56:07.470832+0000","last_peered":"2023-12-07T07:56:07.470832+0000","last_clean":"2023-12-07T07:56:07.470832+0000","last_became_active":"2023-12-07T07:34:47.563950+0000","last_became_peered":"2023-12-07T07:34:47.563950+0000","last_unstale":"2023-12-07T07:56:07.470832+0000","last_undegraded":"2023-12-07T07:56:07.470832+0000","last_fullsized":"2023-12-07T07:56:07.470832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:07.468907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:07.468907+0000","last_clean_scrub_stamp":"2023-12-07T07:56:07.468907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:24:40.285890+0000","scrub_duration":0.023397939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:59.287565+0000","last_change":"2023-12-07T07:55:59.287565+0000","last_active":"2023-12-07T07:55:59.287565+0000","last_peered":"2023-12-07T07:55:59.287565+0000","last_clean":"2023-12-07T07:55:59.287565+0000","last_became_active":"2023-12-07T07:34:47.035305+0000","last_became_peered":"2023-12-07T07:34:47.035305+0000","last_unstale":"2023-12-07T07:55:59.287565+0000","last_undegraded":"2023-12-07T07:55:59.287565+0000","last_fullsized":"2023-12-07T07:55:59.287565+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:59.285437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:59.285437+0000","last_clean_scrub_stamp":"2023-12-07T07:55:59.285437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:24:06.124323+0000","scrub_duration":0.056762805999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":891,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:15.580168+0000","last_change":"2023-12-07T07:56:15.580814+0000","last_active":"2023-12-07T07:56:15.580168+0000","last_peered":"2023-12-07T07:56:15.580168+0000","last_clean":"2023-12-07T07:56:15.580168+0000","last_became_active":"2023-12-07T07:34:55.049584+0000","last_became_peered":"2023-12-07T07:34:55.049584+0000","last_unstale":"2023-12-07T07:56:15.580168+0000","last_undegraded":"2023-12-07T07:56:15.580168+0000","last_fullsized":"2023-12-07T07:56:15.580168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:15.578815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:15.578815+0000","last_clean_scrub_stamp":"2023-12-07T07:56:15.578815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:33:18.224861+0000","scrub_duration":0.025153286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":875,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:53.431731+0000","last_change":"2023-12-07T07:55:53.432421+0000","last_active":"2023-12-07T07:55:53.431731+0000","last_peered":"2023-12-07T07:55:53.431731+0000","last_clean":"2023-12-07T07:55:53.431731+0000","last_became_active":"2023-12-07T07:35:04.480195+0000","last_became_peered":"2023-12-07T07:35:04.480195+0000","last_unstale":"2023-12-07T07:55:53.431731+0000","last_undegraded":"2023-12-07T07:55:53.431731+0000","last_fullsized":"2023-12-07T07:55:53.431731+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:53.430345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:53.430345+0000","last_clean_scrub_stamp":"2023-12-07T07:55:53.430345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:20:02.175037+0000","scrub_duration":0.080473749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":878,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:54.451518+0000","last_change":"2023-12-07T07:55:54.452220+0000","last_active":"2023-12-07T07:55:54.451518+0000","last_peered":"2023-12-07T07:55:54.451518+0000","last_clean":"2023-12-07T07:55:54.451518+0000","last_became_active":"2023-12-07T07:34:51.735923+0000","last_became_peered":"2023-12-07T07:34:51.735923+0000","last_unstale":"2023-12-07T07:55:54.451518+0000","last_undegraded":"2023-12-07T07:55:54.451518+0000","last_fullsized":"2023-12-07T07:55:54.451518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:54.447971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:54.447971+0000","last_clean_scrub_stamp":"2023-12-07T07:55:54.447971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:00:50.135085+0000","scrub_duration":0.041230276000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":900,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:06.482209+0000","last_change":"2023-12-07T07:56:06.482907+0000","last_active":"2023-12-07T07:56:06.482209+0000","last_peered":"2023-12-07T07:56:06.482209+0000","last_clean":"2023-12-07T07:56:06.482209+0000","last_became_active":"2023-12-07T07:34:47.052848+0000","last_became_peered":"2023-12-07T07:34:47.052848+0000","last_unstale":"2023-12-07T07:56:06.482209+0000","last_undegraded":"2023-12-07T07:56:06.482209+0000","last_fullsized":"2023-12-07T07:56:06.482209+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:06.480815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:06.480815+0000","last_clean_scrub_stamp":"2023-12-07T07:56:06.480815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:04:01.002958+0000","scrub_duration":0.027502794000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":888,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:03.309171+0000","last_change":"2023-12-07T07:56:03.309848+0000","last_active":"2023-12-07T07:56:03.309171+0000","last_peered":"2023-12-07T07:56:03.309171+0000","last_clean":"2023-12-07T07:56:03.309171+0000","last_became_active":"2023-12-07T07:34:51.115413+0000","last_became_peered":"2023-12-07T07:34:51.115413+0000","last_unstale":"2023-12-07T07:56:03.309171+0000","last_undegraded":"2023-12-07T07:56:03.309171+0000","last_fullsized":"2023-12-07T07:56:03.309171+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:03.307721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:03.307721+0000","last_clean_scrub_stamp":"2023-12-07T07:56:03.307721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:09:42.334141+0000","scrub_duration":0.033872505999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":901,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:10.553165+0000","last_change":"2023-12-07T07:56:10.553838+0000","last_active":"2023-12-07T07:56:10.553165+0000","last_peered":"2023-12-07T07:56:10.553165+0000","last_clean":"2023-12-07T07:56:10.553165+0000","last_became_active":"2023-12-07T07:34:47.149617+0000","last_became_peered":"2023-12-07T07:34:47.149617+0000","last_unstale":"2023-12-07T07:56:10.553165+0000","last_undegraded":"2023-12-07T07:56:10.553165+0000","last_fullsized":"2023-12-07T07:56:10.553165+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:10.551789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:10.551789+0000","last_clean_scrub_stamp":"2023-12-07T07:56:10.551789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:28:24.694774+0000","scrub_duration":0.066138038999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":892,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:17.653850+0000","last_change":"2023-12-07T07:56:17.654553+0000","last_active":"2023-12-07T07:56:17.653850+0000","last_peered":"2023-12-07T07:56:17.653850+0000","last_clean":"2023-12-07T07:56:17.653850+0000","last_became_active":"2023-12-07T07:34:53.269724+0000","last_became_peered":"2023-12-07T07:34:53.269724+0000","last_unstale":"2023-12-07T07:56:17.653850+0000","last_undegraded":"2023-12-07T07:56:17.653850+0000","last_fullsized":"2023-12-07T07:56:17.653850+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:17.649939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:17.649939+0000","last_clean_scrub_stamp":"2023-12-07T07:56:17.649939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:12:32.883859+0000","scrub_duration":0.040406369999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:05.498663+0000","last_change":"2023-12-07T07:56:05.499436+0000","last_active":"2023-12-07T07:56:05.498663+0000","last_peered":"2023-12-07T07:56:05.498663+0000","last_clean":"2023-12-07T07:56:05.498663+0000","last_became_active":"2023-12-07T07:34:56.746142+0000","last_became_peered":"2023-12-07T07:34:56.746142+0000","last_unstale":"2023-12-07T07:56:05.498663+0000","last_undegraded":"2023-12-07T07:56:05.498663+0000","last_fullsized":"2023-12-07T07:56:05.498663+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:05.497262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:05.497262+0000","last_clean_scrub_stamp":"2023-12-07T07:56:05.497262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:09:06.700261+0000","scrub_duration":0.044731305999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":877,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:59.357476+0000","last_change":"2023-12-07T07:55:59.358134+0000","last_active":"2023-12-07T07:55:59.357476+0000","last_peered":"2023-12-07T07:55:59.357476+0000","last_clean":"2023-12-07T07:55:59.357476+0000","last_became_active":"2023-12-07T07:34:49.356256+0000","last_became_peered":"2023-12-07T07:34:49.356256+0000","last_unstale":"2023-12-07T07:55:59.357476+0000","last_undegraded":"2023-12-07T07:55:59.357476+0000","last_fullsized":"2023-12-07T07:55:59.357476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:59.356120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:59.356120+0000","last_clean_scrub_stamp":"2023-12-07T07:55:59.356120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:29:30.403784+0000","scrub_duration":0.028239416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:05.561765+0000","last_change":"2023-12-07T07:56:05.562470+0000","last_active":"2023-12-07T07:56:05.561765+0000","last_peered":"2023-12-07T07:56:05.561765+0000","last_clean":"2023-12-07T07:56:05.561765+0000","last_became_active":"2023-12-07T07:33:42.665540+0000","last_became_peered":"2023-12-07T07:33:42.665540+0000","last_unstale":"2023-12-07T07:56:05.561765+0000","last_undegraded":"2023-12-07T07:56:05.561765+0000","last_fullsized":"2023-12-07T07:56:05.561765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:05.558570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:05.558570+0000","last_clean_scrub_stamp":"2023-12-07T07:56:05.558570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:46:32.125782+0000","scrub_duration":0.058343892000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:56.535152+0000","last_change":"2023-12-07T07:55:56.536009+0000","last_active":"2023-12-07T07:55:56.535152+0000","last_peered":"2023-12-07T07:55:56.535152+0000","last_clean":"2023-12-07T07:55:56.535152+0000","last_became_active":"2023-12-07T07:34:48.171478+0000","last_became_peered":"2023-12-07T07:34:48.171478+0000","last_unstale":"2023-12-07T07:55:56.535152+0000","last_undegraded":"2023-12-07T07:55:56.535152+0000","last_fullsized":"2023-12-07T07:55:56.535152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:56.533551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:56.533551+0000","last_clean_scrub_stamp":"2023-12-07T07:55:56.533551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:40:56.809166+0000","scrub_duration":0.044084212999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":887,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:56.747016+0000","last_change":"2023-12-07T07:55:56.747748+0000","last_active":"2023-12-07T07:55:56.747016+0000","last_peered":"2023-12-07T07:55:56.747016+0000","last_clean":"2023-12-07T07:55:56.747016+0000","last_became_active":"2023-12-07T07:33:43.190140+0000","last_became_peered":"2023-12-07T07:33:43.190140+0000","last_unstale":"2023-12-07T07:55:56.747016+0000","last_undegraded":"2023-12-07T07:55:56.747016+0000","last_fullsized":"2023-12-07T07:55:56.747016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:56.745587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:56.745587+0000","last_clean_scrub_stamp":"2023-12-07T07:55:56.745587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:42:40.818863+0000","scrub_duration":0.035195790999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":874,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:01.640440+0000","last_change":"2023-12-07T07:56:01.641084+0000","last_active":"2023-12-07T07:56:01.640440+0000","last_peered":"2023-12-07T07:56:01.640440+0000","last_clean":"2023-12-07T07:56:01.640440+0000","last_became_active":"2023-12-07T07:34:49.980936+0000","last_became_peered":"2023-12-07T07:34:49.980936+0000","last_unstale":"2023-12-07T07:56:01.640440+0000","last_undegraded":"2023-12-07T07:56:01.640440+0000","last_fullsized":"2023-12-07T07:56:01.640440+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:01.637388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:01.637388+0000","last_clean_scrub_stamp":"2023-12-07T07:56:01.637388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T19:48:08.865963+0000","scrub_duration":0.059076113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:07.537567+0000","last_change":"2023-12-07T07:56:07.538245+0000","last_active":"2023-12-07T07:56:07.537567+0000","last_peered":"2023-12-07T07:56:07.537567+0000","last_clean":"2023-12-07T07:56:07.537567+0000","last_became_active":"2023-12-07T07:33:42.418871+0000","last_became_peered":"2023-12-07T07:33:42.418871+0000","last_unstale":"2023-12-07T07:56:07.537567+0000","last_undegraded":"2023-12-07T07:56:07.537567+0000","last_fullsized":"2023-12-07T07:56:07.537567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:07.536194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:07.536194+0000","last_clean_scrub_stamp":"2023-12-07T07:56:07.536194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:42:38.367064+0000","scrub_duration":0.026414302000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":900,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.521742+0000","last_change":"2023-12-07T07:56:09.522424+0000","last_active":"2023-12-07T07:56:09.521742+0000","last_peered":"2023-12-07T07:56:09.521742+0000","last_clean":"2023-12-07T07:56:09.521742+0000","last_became_active":"2023-12-07T07:34:51.754214+0000","last_became_peered":"2023-12-07T07:34:51.754214+0000","last_unstale":"2023-12-07T07:56:09.521742+0000","last_undegraded":"2023-12-07T07:56:09.521742+0000","last_fullsized":"2023-12-07T07:56:09.521742+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.520400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.520400+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.520400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:45:14.583169+0000","scrub_duration":0.029549394999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":893,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:55.267024+0000","last_change":"2023-12-07T07:55:55.267692+0000","last_active":"2023-12-07T07:55:55.267024+0000","last_peered":"2023-12-07T07:55:55.267024+0000","last_clean":"2023-12-07T07:55:55.267024+0000","last_became_active":"2023-12-07T07:33:42.628786+0000","last_became_peered":"2023-12-07T07:33:42.628786+0000","last_unstale":"2023-12-07T07:55:55.267024+0000","last_undegraded":"2023-12-07T07:55:55.267024+0000","last_fullsized":"2023-12-07T07:55:55.267024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:55.265638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:55.265638+0000","last_clean_scrub_stamp":"2023-12-07T07:55:55.265638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:51:23.724769+0000","scrub_duration":0.024987489000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":900,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:08.581258+0000","last_change":"2023-12-07T07:56:08.581957+0000","last_active":"2023-12-07T07:56:08.581258+0000","last_peered":"2023-12-07T07:56:08.581258+0000","last_clean":"2023-12-07T07:56:08.581258+0000","last_became_active":"2023-12-07T07:35:04.566145+0000","last_became_peered":"2023-12-07T07:35:04.566145+0000","last_unstale":"2023-12-07T07:56:08.581258+0000","last_undegraded":"2023-12-07T07:56:08.581258+0000","last_fullsized":"2023-12-07T07:56:08.581258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:08.579900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:08.579900+0000","last_clean_scrub_stamp":"2023-12-07T07:56:08.579900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T08:16:11.983753+0000","scrub_duration":0.053430152000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:00.437553+0000","last_change":"2023-12-07T07:56:00.438194+0000","last_active":"2023-12-07T07:56:00.437553+0000","last_peered":"2023-12-07T07:56:00.437553+0000","last_clean":"2023-12-07T07:56:00.437553+0000","last_became_active":"2023-12-07T07:33:42.606012+0000","last_became_peered":"2023-12-07T07:33:42.606012+0000","last_unstale":"2023-12-07T07:56:00.437553+0000","last_undegraded":"2023-12-07T07:56:00.437553+0000","last_fullsized":"2023-12-07T07:56:00.437553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:00.436260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:00.436260+0000","last_clean_scrub_stamp":"2023-12-07T07:56:00.436260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T11:50:56.094814+0000","scrub_duration":0.026847559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":881,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:07.669052+0000","last_change":"2023-12-07T07:56:07.669763+0000","last_active":"2023-12-07T07:56:07.669052+0000","last_peered":"2023-12-07T07:56:07.669052+0000","last_clean":"2023-12-07T07:56:07.669052+0000","last_became_active":"2023-12-07T07:34:47.437980+0000","last_became_peered":"2023-12-07T07:34:47.437980+0000","last_unstale":"2023-12-07T07:56:07.669052+0000","last_undegraded":"2023-12-07T07:56:07.669052+0000","last_fullsized":"2023-12-07T07:56:07.669052+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:07.667590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:07.667590+0000","last_clean_scrub_stamp":"2023-12-07T07:56:07.667590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T12:49:49.966944+0000","scrub_duration":0.066626558000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":906,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:04.636635+0000","last_change":"2023-12-07T07:56:04.636635+0000","last_active":"2023-12-07T07:56:04.636635+0000","last_peered":"2023-12-07T07:56:04.636635+0000","last_clean":"2023-12-07T07:56:04.636635+0000","last_became_active":"2023-12-07T07:33:42.239389+0000","last_became_peered":"2023-12-07T07:33:42.239389+0000","last_unstale":"2023-12-07T07:56:04.636635+0000","last_undegraded":"2023-12-07T07:56:04.636635+0000","last_fullsized":"2023-12-07T07:56:04.636635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:04.628663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:04.628663+0000","last_clean_scrub_stamp":"2023-12-07T07:56:04.628663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:14:45.191025+0000","scrub_duration":0.171417181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":882,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:58.587733+0000","last_change":"2023-12-07T07:55:58.587733+0000","last_active":"2023-12-07T07:55:58.587733+0000","last_peered":"2023-12-07T07:55:58.587733+0000","last_clean":"2023-12-07T07:55:58.587733+0000","last_became_active":"2023-12-07T07:35:04.015545+0000","last_became_peered":"2023-12-07T07:35:04.015545+0000","last_unstale":"2023-12-07T07:55:58.587733+0000","last_undegraded":"2023-12-07T07:55:58.587733+0000","last_fullsized":"2023-12-07T07:55:58.587733+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:58.585592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:58.585592+0000","last_clean_scrub_stamp":"2023-12-07T07:55:58.585592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:15:12.758689+0000","scrub_duration":0.02727336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":897,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:53.453695+0000","last_change":"2023-12-07T07:55:53.453695+0000","last_active":"2023-12-07T07:55:53.453695+0000","last_peered":"2023-12-07T07:55:53.453695+0000","last_clean":"2023-12-07T07:55:53.453695+0000","last_became_active":"2023-12-07T07:33:42.316434+0000","last_became_peered":"2023-12-07T07:33:42.316434+0000","last_unstale":"2023-12-07T07:55:53.453695+0000","last_undegraded":"2023-12-07T07:55:53.453695+0000","last_fullsized":"2023-12-07T07:55:53.453695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:53.451589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:53.451589+0000","last_clean_scrub_stamp":"2023-12-07T07:55:53.451589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T10:55:36.310408+0000","scrub_duration":0.050438442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":886,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:58.797279+0000","last_change":"2023-12-07T07:55:58.797279+0000","last_active":"2023-12-07T07:55:58.797279+0000","last_peered":"2023-12-07T07:55:58.797279+0000","last_clean":"2023-12-07T07:55:58.797279+0000","last_became_active":"2023-12-07T07:34:52.469358+0000","last_became_peered":"2023-12-07T07:34:52.469358+0000","last_unstale":"2023-12-07T07:55:58.797279+0000","last_undegraded":"2023-12-07T07:55:58.797279+0000","last_fullsized":"2023-12-07T07:55:58.797279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:58.794978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:58.794978+0000","last_clean_scrub_stamp":"2023-12-07T07:55:58.794978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:46:26.469454+0000","scrub_duration":0.034781629000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":896,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:04.533048+0000","last_change":"2023-12-07T07:56:04.533683+0000","last_active":"2023-12-07T07:56:04.533048+0000","last_peered":"2023-12-07T07:56:04.533048+0000","last_clean":"2023-12-07T07:56:04.533048+0000","last_became_active":"2023-12-07T07:33:42.644495+0000","last_became_peered":"2023-12-07T07:33:42.644495+0000","last_unstale":"2023-12-07T07:56:04.533048+0000","last_undegraded":"2023-12-07T07:56:04.533048+0000","last_fullsized":"2023-12-07T07:56:04.533048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:04.531736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:04.531736+0000","last_clean_scrub_stamp":"2023-12-07T07:56:04.531736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:34:18.839494+0000","scrub_duration":0.025183835000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"920'78","reported_seq":989,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:52.475342+0000","last_change":"2023-12-07T07:55:52.476595+0000","last_active":"2023-12-07T07:55:52.475342+0000","last_peered":"2023-12-07T07:55:52.475342+0000","last_clean":"2023-12-07T07:55:52.475342+0000","last_became_active":"2023-12-07T07:33:36.349968+0000","last_became_peered":"2023-12-07T07:33:36.349968+0000","last_unstale":"2023-12-07T07:55:52.475342+0000","last_undegraded":"2023-12-07T07:55:52.475342+0000","last_fullsized":"2023-12-07T07:55:52.475342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'78","last_scrub_stamp":"2023-12-07T07:55:52.472086+0000","last_deep_scrub":"920'78","last_deep_scrub_stamp":"2023-12-07T07:55:52.472086+0000","last_clean_scrub_stamp":"2023-12-07T07:55:52.472086+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:50:00.549127+0000","scrub_duration":0.112648125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":889,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.536082+0000","last_change":"2023-12-07T07:56:09.536082+0000","last_active":"2023-12-07T07:56:09.536082+0000","last_peered":"2023-12-07T07:56:09.536082+0000","last_clean":"2023-12-07T07:56:09.536082+0000","last_became_active":"2023-12-07T07:34:47.010508+0000","last_became_peered":"2023-12-07T07:34:47.010508+0000","last_unstale":"2023-12-07T07:56:09.536082+0000","last_undegraded":"2023-12-07T07:56:09.536082+0000","last_fullsized":"2023-12-07T07:56:09.536082+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.530547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.530547+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.530547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:15:04.939748+0000","scrub_duration":0.051766765999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":882,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:08.616961+0000","last_change":"2023-12-07T07:56:08.617643+0000","last_active":"2023-12-07T07:56:08.616961+0000","last_peered":"2023-12-07T07:56:08.616961+0000","last_clean":"2023-12-07T07:56:08.616961+0000","last_became_active":"2023-12-07T07:35:02.498406+0000","last_became_peered":"2023-12-07T07:35:02.498406+0000","last_unstale":"2023-12-07T07:56:08.616961+0000","last_undegraded":"2023-12-07T07:56:08.616961+0000","last_fullsized":"2023-12-07T07:56:08.616961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:08.615636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:08.615636+0000","last_clean_scrub_stamp":"2023-12-07T07:56:08.615636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:23:37.072159+0000","scrub_duration":0.022749836999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1","version":"920'8","reported_seq":890,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:54.579937+0000","last_change":"2023-12-07T07:55:54.581079+0000","last_active":"2023-12-07T07:55:54.579937+0000","last_peered":"2023-12-07T07:55:54.579937+0000","last_clean":"2023-12-07T07:55:54.579937+0000","last_became_active":"2023-12-07T07:34:48.562400+0000","last_became_peered":"2023-12-07T07:34:48.562400+0000","last_unstale":"2023-12-07T07:55:54.579937+0000","last_undegraded":"2023-12-07T07:55:54.579937+0000","last_fullsized":"2023-12-07T07:55:54.579937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'8","last_scrub_stamp":"2023-12-07T07:55:54.577320+0000","last_deep_scrub":"920'8","last_deep_scrub_stamp":"2023-12-07T07:55:54.577320+0000","last_clean_scrub_stamp":"2023-12-07T07:55:54.577320+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T09:40:35.663149+0000","scrub_duration":0.087017894999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":889,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:11.531462+0000","last_change":"2023-12-07T07:56:11.532133+0000","last_active":"2023-12-07T07:56:11.531462+0000","last_peered":"2023-12-07T07:56:11.531462+0000","last_clean":"2023-12-07T07:56:11.531462+0000","last_became_active":"2023-12-07T07:34:50.845529+0000","last_became_peered":"2023-12-07T07:34:50.845529+0000","last_unstale":"2023-12-07T07:56:11.531462+0000","last_undegraded":"2023-12-07T07:56:11.531462+0000","last_fullsized":"2023-12-07T07:56:11.531462+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:11.529980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:11.529980+0000","last_clean_scrub_stamp":"2023-12-07T07:56:11.529980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:15:52.318349+0000","scrub_duration":0.033477287000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":892,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.608617+0000","last_change":"2023-12-07T07:56:09.609295+0000","last_active":"2023-12-07T07:56:09.608617+0000","last_peered":"2023-12-07T07:56:09.608617+0000","last_clean":"2023-12-07T07:56:09.608617+0000","last_became_active":"2023-12-07T07:34:44.899627+0000","last_became_peered":"2023-12-07T07:34:44.899627+0000","last_unstale":"2023-12-07T07:56:09.608617+0000","last_undegraded":"2023-12-07T07:56:09.608617+0000","last_fullsized":"2023-12-07T07:56:09.608617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.605252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.605252+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.605252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:30:15.455205+0000","scrub_duration":0.053693215000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":876,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:58.522399+0000","last_change":"2023-12-07T07:55:58.523101+0000","last_active":"2023-12-07T07:55:58.522399+0000","last_peered":"2023-12-07T07:55:58.522399+0000","last_clean":"2023-12-07T07:55:58.522399+0000","last_became_active":"2023-12-07T07:34:50.418281+0000","last_became_peered":"2023-12-07T07:34:50.418281+0000","last_unstale":"2023-12-07T07:55:58.522399+0000","last_undegraded":"2023-12-07T07:55:58.522399+0000","last_fullsized":"2023-12-07T07:55:58.522399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:58.518343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:58.518343+0000","last_clean_scrub_stamp":"2023-12-07T07:55:58.518343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T18:41:54.911463+0000","scrub_duration":0.024849433000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":876,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:54.356961+0000","last_change":"2023-12-07T07:55:54.357655+0000","last_active":"2023-12-07T07:55:54.356961+0000","last_peered":"2023-12-07T07:55:54.356961+0000","last_clean":"2023-12-07T07:55:54.356961+0000","last_became_active":"2023-12-07T07:34:58.263210+0000","last_became_peered":"2023-12-07T07:34:58.263210+0000","last_unstale":"2023-12-07T07:55:54.356961+0000","last_undegraded":"2023-12-07T07:55:54.356961+0000","last_fullsized":"2023-12-07T07:55:54.356961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:54.355609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:54.355609+0000","last_clean_scrub_stamp":"2023-12-07T07:55:54.355609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T16:31:59.519974+0000","scrub_duration":0.046963756000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":889,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:19.451918+0000","last_change":"2023-12-07T07:56:19.451918+0000","last_active":"2023-12-07T07:56:19.451918+0000","last_peered":"2023-12-07T07:56:19.451918+0000","last_clean":"2023-12-07T07:56:19.451918+0000","last_became_active":"2023-12-07T07:34:47.225369+0000","last_became_peered":"2023-12-07T07:34:47.225369+0000","last_unstale":"2023-12-07T07:56:19.451918+0000","last_undegraded":"2023-12-07T07:56:19.451918+0000","last_fullsized":"2023-12-07T07:56:19.451918+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:19.449801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:19.449801+0000","last_clean_scrub_stamp":"2023-12-07T07:56:19.449801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T16:40:38.819291+0000","scrub_duration":0.041672700999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":876,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:09.472800+0000","last_change":"2023-12-07T07:56:09.472800+0000","last_active":"2023-12-07T07:56:09.472800+0000","last_peered":"2023-12-07T07:56:09.472800+0000","last_clean":"2023-12-07T07:56:09.472800+0000","last_became_active":"2023-12-07T07:35:00.363892+0000","last_became_peered":"2023-12-07T07:35:00.363892+0000","last_unstale":"2023-12-07T07:56:09.472800+0000","last_undegraded":"2023-12-07T07:56:09.472800+0000","last_fullsized":"2023-12-07T07:56:09.472800+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:09.470782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:09.470782+0000","last_clean_scrub_stamp":"2023-12-07T07:56:09.470782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:16:57.692809+0000","scrub_duration":0.063907027000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":881,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:57.548535+0000","last_change":"2023-12-07T07:55:57.548535+0000","last_active":"2023-12-07T07:55:57.548535+0000","last_peered":"2023-12-07T07:55:57.548535+0000","last_clean":"2023-12-07T07:55:57.548535+0000","last_became_active":"2023-12-07T07:34:45.533595+0000","last_became_peered":"2023-12-07T07:34:45.533595+0000","last_unstale":"2023-12-07T07:55:57.548535+0000","last_undegraded":"2023-12-07T07:55:57.548535+0000","last_fullsized":"2023-12-07T07:55:57.548535+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:57.546560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:57.546560+0000","last_clean_scrub_stamp":"2023-12-07T07:55:57.546560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T13:01:03.889741+0000","scrub_duration":0.024617533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":885,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:57.796234+0000","last_change":"2023-12-07T07:55:57.796234+0000","last_active":"2023-12-07T07:55:57.796234+0000","last_peered":"2023-12-07T07:55:57.796234+0000","last_clean":"2023-12-07T07:55:57.796234+0000","last_became_active":"2023-12-07T07:34:54.918404+0000","last_became_peered":"2023-12-07T07:34:54.918404+0000","last_unstale":"2023-12-07T07:55:57.796234+0000","last_undegraded":"2023-12-07T07:55:57.796234+0000","last_fullsized":"2023-12-07T07:55:57.796234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:57.794282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:57.794282+0000","last_clean_scrub_stamp":"2023-12-07T07:55:57.794282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T14:32:29.043381+0000","scrub_duration":0.041595646999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":890,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:56:13.545920+0000","last_change":"2023-12-07T07:56:13.546610+0000","last_active":"2023-12-07T07:56:13.545920+0000","last_peered":"2023-12-07T07:56:13.545920+0000","last_clean":"2023-12-07T07:56:13.545920+0000","last_became_active":"2023-12-07T07:34:45.408105+0000","last_became_peered":"2023-12-07T07:34:45.408105+0000","last_unstale":"2023-12-07T07:56:13.545920+0000","last_undegraded":"2023-12-07T07:56:13.545920+0000","last_fullsized":"2023-12-07T07:56:13.545920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:56:13.544607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:56:13.544607+0000","last_clean_scrub_stamp":"2023-12-07T07:56:13.544607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T17:49:50.853888+0000","scrub_duration":0.026673565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":877,"reported_epoch":920,"state":"active+clean","last_fresh":"2023-12-07T07:55:55.492980+0000","last_change":"2023-12-07T07:55:55.493639+0000","last_active":"2023-12-07T07:55:55.492980+0000","last_peered":"2023-12-07T07:55:55.492980+0000","last_clean":"2023-12-07T07:55:55.492980+0000","last_became_active":"2023-12-07T07:34:49.378907+0000","last_became_peered":"2023-12-07T07:34:49.378907+0000","last_unstale":"2023-12-07T07:55:55.492980+0000","last_undegraded":"2023-12-07T07:55:55.492980+0000","last_fullsized":"2023-12-07T07:55:55.492980+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T07:55:55.491658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T07:55:55.491658+0000","last_clean_scrub_stamp":"2023-12-07T07:55:55.491658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-08T15:28:12.211489+0000","scrub_duration":0.058683120999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444309,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:56:15 2023","interfaces":[{"interface":"back","average":{"1min":14.324999999999999,"5min":23.972000000000001,"15min":27.324999999999999},"min":{"1min":5.9219999999999997,"5min":5.7389999999999999,"15min":3.8220000000000001},"max":{"1min":100.182,"5min":240.48099999999999,"15min":963.553},"last":14.920999999999999},{"interface":"front","average":{"1min":13.577,"5min":28.972000000000001,"15min":23.814},"min":{"1min":5.976,"5min":5.976,"15min":4.7640000000000002},"max":{"1min":23.638999999999999,"5min":252.37899999999999,"15min":285.09399999999999},"last":16.227}]},{"osd":1,"last update":"Thu Dec 7 07:56:15 2023","interfaces":[{"interface":"back","average":{"1min":10.128,"5min":19.628,"15min":17.001999999999999},"min":{"1min":6.3440000000000003,"5min":5.8899999999999997,"15min":5.5049999999999999},"max":{"1min":23.829000000000001,"5min":236.608,"15min":236.608},"last":8.9450000000000003},{"interface":"front","average":{"1min":13.397,"5min":27.309000000000001,"15min":21.978000000000002},"min":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":6.3490000000000002},"max":{"1min":25.346,"5min":275.33300000000003,"15min":275.33300000000003},"last":12.705}]},{"osd":2,"last update":"Thu Dec 7 07:56:15 2023","interfaces":[{"interface":"back","average":{"1min":14.789999999999999,"5min":27.204000000000001,"15min":22.213999999999999},"min":{"1min":10.228999999999999,"5min":10.228999999999999,"15min":7.4749999999999996},"max":{"1min":21.253,"5min":277.43099999999998,"15min":277.43099999999998},"last":8.0549999999999997},{"interface":"front","average":{"1min":10.762,"5min":21.241,"15min":22.661000000000001},"min":{"1min":6.423,"5min":6.3700000000000001,"15min":6.3700000000000001},"max":{"1min":17.742999999999999,"5min":243.97800000000001,"15min":243.97800000000001},"last":9.6270000000000007}]},{"osd":4,"last update":"Thu Dec 7 07:56:15 2023","interfaces":[{"interface":"back","average":{"1min":9.5960000000000001,"5min":20.135000000000002,"15min":19.119},"min":{"1min":5.992,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":24.079999999999998,"5min":195.59299999999999,"15min":195.59299999999999},"last":7.4340000000000002},{"interface":"front","average":{"1min":13.458,"5min":25.503,"15min":23.702999999999999},"min":{"1min":7.9889999999999999,"5min":7.9889999999999999,"15min":7.1500000000000004},"max":{"1min":18.901,"5min":215.51900000000001,"15min":242.465},"last":10.199}]},{"osd":5,"last update":"Thu Dec 7 07:56:25 2023","interfaces":[{"interface":"back","average":{"1min":14.487,"5min":26.175999999999998,"15min":25.581},"min":{"1min":9.2669999999999995,"5min":7.2640000000000002,"15min":7.2640000000000002},"max":{"1min":25.742999999999999,"5min":241.82300000000001,"15min":241.82300000000001},"last":15.553000000000001},{"interface":"front","average":{"1min":18.238,"5min":29.029,"15min":27.745000000000001},"min":{"1min":13.179,"5min":9.3460000000000001,"15min":8.6780000000000008},"max":{"1min":26.216000000000001,"5min":286.17200000000003,"15min":286.17200000000003},"last":17.449000000000002}]},{"osd":6,"last update":"Thu Dec 7 07:56:15 2023","interfaces":[{"interface":"back","average":{"1min":11.398999999999999,"5min":18.454999999999998,"15min":22.475999999999999},"min":{"1min":7.8860000000000001,"5min":6.149,"15min":6.149},"max":{"1min":18.321000000000002,"5min":168.52600000000001,"15min":210.505},"last":11.395},{"interface":"front","average":{"1min":17.038,"5min":26.155000000000001,"15min":25.327000000000002},"min":{"1min":10.272,"5min":10.272,"15min":9.0559999999999992},"max":{"1min":24.472000000000001,"5min":168.88499999999999,"15min":168.88499999999999},"last":13.446999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:56:15 2023","interfaces":[{"interface":"back","average":{"1min":18.027999999999999,"5min":25.684999999999999,"15min":26.638999999999999},"min":{"1min":12.944000000000001,"5min":8.9139999999999997,"15min":8.0549999999999997},"max":{"1min":25.364000000000001,"5min":186.81299999999999,"15min":247.31100000000001},"last":18.260999999999999},{"interface":"front","average":{"1min":13.199,"5min":19.687000000000001,"15min":22.385999999999999},"min":{"1min":8.3620000000000001,"5min":6.8410000000000002,"15min":6.8410000000000002},"max":{"1min":29.184000000000001,"5min":184.453,"15min":311.41000000000003},"last":12.019}]}]},{"osd":2,"up_from":17,"seq":73014444309,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64120,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94307720,"statfs":{"total":96636764160,"available":96571105280,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":16.497,"5min":15.779999999999999,"15min":16.457000000000001},"min":{"1min":6.3470000000000004,"5min":5.9960000000000004,"15min":4.9409999999999998},"max":{"1min":172.768,"5min":172.768,"15min":339.37299999999999},"last":7.4450000000000003},{"interface":"front","average":{"1min":19.518999999999998,"5min":20.863,"15min":23.408000000000001},"min":{"1min":7.7599999999999998,"5min":7.0949999999999998,"15min":4.6319999999999997},"max":{"1min":174.06200000000001,"5min":174.06200000000001,"15min":597.92399999999998},"last":8.7590000000000003}]},{"osd":1,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":20.234000000000002,"5min":19.273,"15min":19.411000000000001},"min":{"1min":9.8040000000000003,"5min":7.4930000000000003,"15min":6.1879999999999997},"max":{"1min":158.005,"5min":158.005,"15min":235.55600000000001},"last":12.369999999999999},{"interface":"front","average":{"1min":13.27,"5min":14.262,"15min":16.658000000000001},"min":{"1min":5.3819999999999997,"5min":5.093,"15min":5.093},"max":{"1min":111.26600000000001,"5min":111.26600000000001,"15min":135.71000000000001},"last":6.6500000000000004}]},{"osd":3,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":21.927,"5min":24.766999999999999,"15min":20.167000000000002},"min":{"1min":11.124000000000001,"5min":8.1869999999999994,"15min":6.1699999999999999},"max":{"1min":144.161,"5min":160.94800000000001,"15min":160.94800000000001},"last":13.045999999999999},{"interface":"front","average":{"1min":16.93,"5min":19.02,"15min":19.995000000000001},"min":{"1min":7.2930000000000001,"5min":6.5739999999999998,"15min":6.5170000000000003},"max":{"1min":89.558000000000007,"5min":128.06100000000001,"15min":136.256},"last":11.090999999999999}]},{"osd":4,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":20.617000000000001,"5min":19.425000000000001,"15min":19.09},"min":{"1min":7.6849999999999996,"5min":7.0599999999999996,"15min":6.5570000000000004},"max":{"1min":90.582999999999998,"5min":130.738,"15min":229.14500000000001},"last":17.202000000000002},{"interface":"front","average":{"1min":25.648,"5min":23.628,"15min":22.283000000000001},"min":{"1min":9.7420000000000009,"5min":9.7420000000000009,"15min":6.1550000000000002},"max":{"1min":165.47999999999999,"5min":165.47999999999999,"15min":284.95100000000002},"last":13.673}]},{"osd":5,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":15.595000000000001,"5min":17.318000000000001,"15min":16.949000000000002},"min":{"1min":6.1390000000000002,"5min":6.0019999999999998,"15min":5.9489999999999998},"max":{"1min":131.90899999999999,"5min":144.864,"15min":247.37799999999999},"last":10.003},{"interface":"front","average":{"1min":21.292000000000002,"5min":22.408000000000001,"15min":21.466000000000001},"min":{"1min":11.361000000000001,"5min":6.7229999999999999,"15min":6.7229999999999999},"max":{"1min":112.038,"5min":143.517,"15min":218.82400000000001},"last":18.532}]},{"osd":6,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":23.312000000000001,"5min":24.370000000000001,"15min":25.202000000000002},"min":{"1min":12.407999999999999,"5min":9.5549999999999997,"15min":7.7190000000000003},"max":{"1min":120.989,"5min":133.203,"15min":288.38299999999998},"last":19.617000000000001},{"interface":"front","average":{"1min":25.363,"5min":26.001999999999999,"15min":27.93},"min":{"1min":13.728999999999999,"5min":10.220000000000001,"15min":8.25},"max":{"1min":166.917,"5min":182.745,"15min":291.435},"last":14.343999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":21.131,"5min":20.477,"15min":22.096},"min":{"1min":7.0090000000000003,"5min":6.4740000000000002,"15min":6.4740000000000002},"max":{"1min":120.267,"5min":162.23500000000001,"15min":241.679},"last":16.757000000000001},{"interface":"front","average":{"1min":27.395,"5min":26.259,"15min":23.024999999999999},"min":{"1min":14.006,"5min":10.09,"15min":5.7859999999999996},"max":{"1min":132.678,"5min":136.339,"15min":249.32499999999999},"last":20.719000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444310,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60356,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311484,"statfs":{"total":96636764160,"available":96574959616,"internally_reserved":0,"allocated":7139328,"data_stored":4075528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:56:12 2023","interfaces":[{"interface":"back","average":{"1min":20.143999999999998,"5min":25.202000000000002,"15min":22.259},"min":{"1min":5.3079999999999998,"5min":4.3029999999999999,"15min":4.2080000000000002},"max":{"1min":170.88200000000001,"5min":348.17200000000003,"15min":410.08199999999999},"last":6.6079999999999997},{"interface":"front","average":{"1min":20.643000000000001,"5min":20.329000000000001,"15min":19.542000000000002},"min":{"1min":5.141,"5min":5.141,"15min":5.141},"max":{"1min":123.119,"5min":146.68700000000001,"15min":303.291},"last":8.4770000000000003}]},{"osd":2,"last update":"Thu Dec 7 07:56:12 2023","interfaces":[{"interface":"back","average":{"1min":23.681999999999999,"5min":21.427,"15min":21.542999999999999},"min":{"1min":6.7190000000000003,"5min":4.4240000000000004,"15min":4.4240000000000004},"max":{"1min":183.63399999999999,"5min":192.226,"15min":336.255},"last":7.6900000000000004},{"interface":"front","average":{"1min":25.538,"5min":28.379999999999999,"15min":22.155999999999999},"min":{"1min":8.8670000000000009,"5min":7.2610000000000001,"15min":5.2679999999999998},"max":{"1min":171.90299999999999,"5min":229.56100000000001,"15min":335.88499999999999},"last":12.573}]},{"osd":3,"last update":"Thu Dec 7 07:56:12 2023","interfaces":[{"interface":"back","average":{"1min":27.596,"5min":28.507000000000001,"15min":24.652000000000001},"min":{"1min":6.8810000000000002,"5min":6.8810000000000002,"15min":6.3940000000000001},"max":{"1min":175.78800000000001,"5min":317.94999999999999,"15min":429.30399999999997},"last":9.4510000000000005},{"interface":"front","average":{"1min":19.544,"5min":29.786999999999999,"15min":23.616},"min":{"1min":6.54,"5min":6.54,"15min":6.1589999999999998},"max":{"1min":106.224,"5min":302.00599999999997,"15min":382.79399999999998},"last":17.027000000000001}]},{"osd":4,"last update":"Thu Dec 7 07:56:12 2023","interfaces":[{"interface":"back","average":{"1min":28.033000000000001,"5min":29.228999999999999,"15min":27.959},"min":{"1min":9.8320000000000007,"5min":8.7759999999999998,"15min":5.9969999999999999},"max":{"1min":159.37,"5min":159.37,"15min":406.53899999999999},"last":14.353999999999999},{"interface":"front","average":{"1min":25.317,"5min":33.148000000000003,"15min":27.565999999999999},"min":{"1min":7.2519999999999998,"5min":7.2519999999999998,"15min":6.9279999999999999},"max":{"1min":180.47,"5min":417.81700000000001,"15min":417.81700000000001},"last":10.055}]},{"osd":5,"last update":"Thu Dec 7 07:56:12 2023","interfaces":[{"interface":"back","average":{"1min":23.931000000000001,"5min":27.216999999999999,"15min":26.574000000000002},"min":{"1min":8.532,"5min":8.532,"15min":7.3049999999999997},"max":{"1min":118.05200000000001,"5min":128.53899999999999,"15min":390.11000000000001},"last":15.416},{"interface":"front","average":{"1min":26.652000000000001,"5min":32.965000000000003,"15min":26.513999999999999},"min":{"1min":12.119999999999999,"5min":8.8249999999999993,"15min":7.0350000000000001},"max":{"1min":172.87700000000001,"5min":323.339,"15min":357.44799999999998},"last":10.773999999999999}]},{"osd":6,"last update":"Thu Dec 7 07:56:12 2023","interfaces":[{"interface":"back","average":{"1min":23.279,"5min":29.300999999999998,"15min":30.061},"min":{"1min":10.164,"5min":10.164,"15min":7.9279999999999999},"max":{"1min":112.759,"5min":188.14400000000001,"15min":420.565},"last":11.879},{"interface":"front","average":{"1min":28.452999999999999,"5min":34.670000000000002,"15min":31.27},"min":{"1min":11.882999999999999,"5min":11.882999999999999,"15min":8.8680000000000003},"max":{"1min":117.053,"5min":264.226,"15min":340.71100000000001},"last":16.341000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:56:12 2023","interfaces":[{"interface":"back","average":{"1min":16.966000000000001,"5min":23.116,"15min":25.934000000000001},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":92.885999999999996,"5min":200.01900000000001,"15min":442.14100000000002},"last":14.064},{"interface":"front","average":{"1min":26.864000000000001,"5min":31.451000000000001,"15min":29.329000000000001},"min":{"1min":13.867000000000001,"5min":12.192,"15min":8.8699999999999992},"max":{"1min":118.66200000000001,"5min":267.07999999999998,"15min":533.87199999999996},"last":17.952000000000002}]}]},{"osd":0,"up_from":16,"seq":68719477013,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59964,"kb_used_data":6580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311876,"statfs":{"total":96636764160,"available":96575361024,"internally_reserved":0,"allocated":6737920,"data_stored":3677688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":13.933,"5min":19.175999999999998,"15min":18.600999999999999},"min":{"1min":6.758,"5min":5.3540000000000001,"15min":5.3529999999999998},"max":{"1min":39.606999999999999,"5min":230.792,"15min":230.792},"last":14.159000000000001},{"interface":"front","average":{"1min":16.013999999999999,"5min":20.643000000000001,"15min":23.875},"min":{"1min":10.679,"5min":6.9210000000000003,"15min":6.9210000000000003},"max":{"1min":40.453000000000003,"5min":105.54900000000001,"15min":273.524},"last":14.984}]},{"osd":2,"last update":"Thu Dec 7 07:56:14 2023","interfaces":[{"interface":"back","average":{"1min":15.904999999999999,"5min":22.483000000000001,"15min":22.661000000000001},"min":{"1min":9.4640000000000004,"5min":8.8759999999999994,"15min":8.1080000000000005},"max":{"1min":41.075000000000003,"5min":201.53800000000001,"15min":201.53800000000001},"last":13.262},{"interface":"front","average":{"1min":11.013,"5min":14.693,"15min":15.558999999999999},"min":{"1min":4.8499999999999996,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":34.811,"5min":152.64699999999999,"15min":214.94},"last":6.5839999999999996}]},{"osd":3,"last update":"Thu Dec 7 07:56:14 2023","interfaces":[{"interface":"back","average":{"1min":10.911,"5min":20.151,"15min":24.960000000000001},"min":{"1min":6.3369999999999997,"5min":5.3440000000000003,"15min":5.0899999999999999},"max":{"1min":38.741999999999997,"5min":283.43000000000001,"15min":932.75999999999999},"last":9.3379999999999992},{"interface":"front","average":{"1min":17.753,"5min":24.341000000000001,"15min":25.675000000000001},"min":{"1min":11.99,"5min":11.481,"15min":8.0809999999999995},"max":{"1min":41.843000000000004,"5min":256.267,"15min":300.13999999999999},"last":15.821}]},{"osd":4,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":30.655999999999999,"5min":23.756,"15min":28.148},"min":{"1min":8.4250000000000007,"5min":6.7930000000000001,"15min":6.4649999999999999},"max":{"1min":226.23400000000001,"5min":226.23400000000001,"15min":1049.731},"last":10.676},{"interface":"front","average":{"1min":29.693000000000001,"5min":28.751999999999999,"15min":27.335000000000001},"min":{"1min":15.347,"5min":13.824,"15min":9.5399999999999991},"max":{"1min":130.922,"5min":158.40100000000001,"15min":283.43700000000001},"last":16.628}]},{"osd":5,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":25.518000000000001,"5min":29.251000000000001,"15min":28.359999999999999},"min":{"1min":11.894,"5min":10.550000000000001,"15min":5.8300000000000001},"max":{"1min":87.106999999999999,"5min":248.34299999999999,"15min":255.328},"last":17.459},{"interface":"front","average":{"1min":24.747,"5min":23.076000000000001,"15min":29.238},"min":{"1min":7.7629999999999999,"5min":6.4509999999999996,"15min":5.7110000000000003},"max":{"1min":142.75700000000001,"5min":264.34899999999999,"15min":1217.117},"last":11.396000000000001}]},{"osd":6,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":19.439,"15min":28.902000000000001},"min":{"1min":9.0489999999999995,"5min":7.8789999999999996,"15min":7.0350000000000001},"max":{"1min":45.210000000000001,"5min":154.68199999999999,"15min":1252.499},"last":10.106999999999999},{"interface":"front","average":{"1min":20.082000000000001,"5min":28.024999999999999,"15min":29.802},"min":{"1min":13.231999999999999,"5min":13.167999999999999,"15min":11.638},"max":{"1min":46.444000000000003,"5min":263.43200000000002,"15min":263.43200000000002},"last":18.381}]},{"osd":7,"last update":"Thu Dec 7 07:56:20 2023","interfaces":[{"interface":"back","average":{"1min":33.161999999999999,"5min":29.91,"15min":31.620999999999999},"min":{"1min":16.114000000000001,"5min":13.768000000000001,"15min":13.212},"max":{"1min":138.602,"5min":138.602,"15min":279.38900000000001},"last":19.219000000000001},{"interface":"front","average":{"1min":33.066000000000003,"5min":25.148,"15min":23.571999999999999},"min":{"1min":7.4130000000000003,"5min":7.1600000000000001,"15min":6.2990000000000004},"max":{"1min":215.11699999999999,"5min":262.07299999999998,"15min":262.07299999999998},"last":8.3140000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542422,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59960,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311880,"statfs":{"total":96636764160,"available":96575365120,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:56:29 2023","interfaces":[{"interface":"back","average":{"1min":13.52,"5min":16.190000000000001,"15min":16.844999999999999},"min":{"1min":6.0890000000000004,"5min":5.4500000000000002,"15min":4.8579999999999997},"max":{"1min":59.363999999999997,"5min":227.49000000000001,"15min":227.49000000000001},"last":13.519},{"interface":"front","average":{"1min":18.724,"5min":21.962,"15min":22.824999999999999},"min":{"1min":8.218,"5min":6.343,"15min":5.9909999999999997},"max":{"1min":69.679000000000002,"5min":228.13399999999999,"15min":228.13399999999999},"last":18.690999999999999}]},{"osd":1,"last update":"Thu Dec 7 07:56:29 2023","interfaces":[{"interface":"back","average":{"1min":18.768000000000001,"5min":20.353000000000002,"15min":21.289000000000001},"min":{"1min":9.5030000000000001,"5min":6.2599999999999998,"15min":5.9390000000000001},"max":{"1min":64.655000000000001,"5min":228.745,"15min":228.745},"last":17.794},{"interface":"front","average":{"1min":25.66,"5min":19,"15min":17.734000000000002},"min":{"1min":4.7160000000000002,"5min":4.7160000000000002,"15min":4.6100000000000003},"max":{"1min":244.995,"5min":244.995,"15min":244.995},"last":9.7390000000000008}]},{"osd":2,"last update":"Thu Dec 7 07:56:16 2023","interfaces":[{"interface":"back","average":{"1min":19.504999999999999,"5min":19.873999999999999,"15min":17.673999999999999},"min":{"1min":7.0049999999999999,"5min":5.8680000000000003,"15min":4.4210000000000003},"max":{"1min":108.685,"5min":156.13499999999999,"15min":215.05699999999999},"last":8.3059999999999992},{"interface":"front","average":{"1min":20.904,"5min":25.835000000000001,"15min":24.308},"min":{"1min":7.8040000000000003,"5min":7.8040000000000003,"15min":7.8040000000000003},"max":{"1min":68.046999999999997,"5min":232.45400000000001,"15min":232.45400000000001},"last":16.891999999999999}]},{"osd":3,"last update":"Thu Dec 7 07:56:29 2023","interfaces":[{"interface":"back","average":{"1min":35.783000000000001,"5min":28.803000000000001,"15min":22.995999999999999},"min":{"1min":7.8079999999999998,"5min":6.024,"15min":6.024},"max":{"1min":245.643,"5min":248.97900000000001,"15min":248.97900000000001},"last":11.175000000000001},{"interface":"front","average":{"1min":25.962,"5min":27.294,"15min":22.327999999999999},"min":{"1min":9.0090000000000003,"5min":6.9790000000000001,"15min":6.0860000000000003},"max":{"1min":149.57499999999999,"5min":236.92699999999999,"15min":236.92699999999999},"last":9.0600000000000005}]},{"osd":4,"last update":"Thu Dec 7 07:56:00 2023","interfaces":[{"interface":"back","average":{"1min":35.658000000000001,"5min":30.891999999999999,"15min":24.102},"min":{"1min":8.6880000000000006,"5min":6.3899999999999997,"15min":6.3090000000000002},"max":{"1min":246.28299999999999,"5min":276.92500000000001,"15min":276.92500000000001},"last":12.727},{"interface":"front","average":{"1min":17.623999999999999,"5min":27.646000000000001,"15min":23.425000000000001},"min":{"1min":6.7910000000000004,"5min":6.7910000000000004,"15min":6.7910000000000004},"max":{"1min":47.604999999999997,"5min":179.64099999999999,"15min":220.881},"last":11.948}]},{"osd":5,"last update":"Thu Dec 7 07:56:00 2023","interfaces":[{"interface":"back","average":{"1min":30.048999999999999,"5min":26.992000000000001,"15min":25.532},"min":{"1min":7.9720000000000004,"5min":7.8570000000000002,"15min":7.8570000000000002},"max":{"1min":152.00399999999999,"5min":237.53700000000001,"15min":237.53700000000001},"last":15.058999999999999},{"interface":"front","average":{"1min":28.152999999999999,"5min":28.809000000000001,"15min":25.042999999999999},"min":{"1min":8.6590000000000007,"5min":8.6590000000000007,"15min":8.0099999999999998},"max":{"1min":127.7,"5min":239.673,"15min":239.673},"last":19.577000000000002}]},{"osd":7,"last update":"Thu Dec 7 07:56:16 2023","interfaces":[{"interface":"back","average":{"1min":24.986000000000001,"5min":28.611999999999998,"15min":25.402000000000001},"min":{"1min":8.9909999999999997,"5min":8.4890000000000008,"15min":7.7670000000000003},"max":{"1min":100.85599999999999,"5min":236.26499999999999,"15min":236.26499999999999},"last":15.696},{"interface":"front","average":{"1min":24.539999999999999,"5min":29.484000000000002,"15min":28.277000000000001},"min":{"1min":13.042999999999999,"5min":10.239000000000001,"15min":10.239000000000001},"max":{"1min":96.189999999999998,"5min":253.29300000000001,"15min":253.29300000000001},"last":20.414999999999999}]}]},{"osd":7,"up_from":13,"seq":55834575128,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78008,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293832,"statfs":{"total":96636764160,"available":96556883968,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:56:19 2023","interfaces":[{"interface":"back","average":{"1min":11.06,"5min":10.513999999999999,"15min":15.215999999999999},"min":{"1min":5.0300000000000002,"5min":4.3899999999999997,"15min":4.29},"max":{"1min":63.902999999999999,"5min":63.902999999999999,"15min":214.32599999999999},"last":9.9350000000000005},{"interface":"front","average":{"1min":10.228999999999999,"5min":9.7940000000000005,"15min":14.44},"min":{"1min":4.5179999999999998,"5min":4.3659999999999997,"15min":4.3659999999999997},"max":{"1min":55.290999999999997,"5min":55.290999999999997,"15min":199.94499999999999},"last":7.633}]},{"osd":1,"last update":"Thu Dec 7 07:56:19 2023","interfaces":[{"interface":"back","average":{"1min":11.109999999999999,"5min":14.779,"15min":18.381},"min":{"1min":5.274,"5min":5.274,"15min":5.274},"max":{"1min":21.419,"5min":117.795,"15min":198.512},"last":16.155000000000001},{"interface":"front","average":{"1min":13.922000000000001,"5min":15.707000000000001,"15min":19.231000000000002},"min":{"1min":6.9870000000000001,"5min":5.7270000000000003,"15min":5.6929999999999996},"max":{"1min":63.003999999999998,"5min":119.175,"15min":167.78200000000001},"last":17.866}]},{"osd":2,"last update":"Thu Dec 7 07:56:19 2023","interfaces":[{"interface":"back","average":{"1min":13.912000000000001,"5min":16.07,"15min":17.84},"min":{"1min":7.4130000000000003,"5min":6.0049999999999999,"15min":5.9669999999999996},"max":{"1min":65.513000000000005,"5min":253.48699999999999,"15min":282.12099999999998},"last":12.087999999999999},{"interface":"front","average":{"1min":14.864000000000001,"5min":16.113,"15min":19.206},"min":{"1min":7.8179999999999996,"5min":6.9939999999999998,"15min":6.9939999999999998},"max":{"1min":51.826000000000001,"5min":63.570999999999998,"15min":216.28299999999999},"last":16.994}]},{"osd":3,"last update":"Thu Dec 7 07:56:19 2023","interfaces":[{"interface":"back","average":{"1min":16.52,"5min":17.166,"15min":19.236000000000001},"min":{"1min":5.8659999999999997,"5min":5.8659999999999997,"15min":5.8659999999999997},"max":{"1min":78.504999999999995,"5min":84.263000000000005,"15min":159.934},"last":15.002000000000001},{"interface":"front","average":{"1min":18.876000000000001,"5min":18.899999999999999,"15min":19.594999999999999},"min":{"1min":8.7759999999999998,"5min":7.2309999999999999,"15min":6.9409999999999998},"max":{"1min":123.455,"5min":147.85400000000001,"15min":157.43100000000001},"last":12.802}]},{"osd":4,"last update":"Thu Dec 7 07:56:19 2023","interfaces":[{"interface":"back","average":{"1min":16.491,"5min":17.245000000000001,"15min":21.396000000000001},"min":{"1min":10.904999999999999,"5min":7.2110000000000003,"15min":6.7000000000000002},"max":{"1min":56.920000000000002,"5min":68.998999999999995,"15min":204.71199999999999},"last":18.702000000000002},{"interface":"front","average":{"1min":17.731000000000002,"5min":18.056000000000001,"15min":20.725999999999999},"min":{"1min":9.7899999999999991,"5min":7.7839999999999998,"15min":7.1269999999999998},"max":{"1min":61.832000000000001,"5min":76.503,"15min":259.23200000000003},"last":14.128}]},{"osd":5,"last update":"Thu Dec 7 07:56:19 2023","interfaces":[{"interface":"back","average":{"1min":19.968,"5min":22.443000000000001,"15min":24.492000000000001},"min":{"1min":9.8130000000000006,"5min":9.2409999999999997,"15min":8.9830000000000005},"max":{"1min":107.327,"5min":189.49000000000001,"15min":239.32599999999999},"last":19.567},{"interface":"front","average":{"1min":19.224,"5min":19.489999999999998,"15min":22.664999999999999},"min":{"1min":11.124000000000001,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":93.210999999999999,"5min":157.60400000000001,"15min":216.947},"last":10.695}]},{"osd":6,"last update":"Thu Dec 7 07:56:19 2023","interfaces":[{"interface":"back","average":{"1min":18.84,"5min":20.346,"15min":23.797000000000001},"min":{"1min":11.522,"5min":9.9390000000000001,"15min":8.2750000000000004},"max":{"1min":58.777999999999999,"5min":66.572999999999993,"15min":192.76300000000001},"last":20.405999999999999},{"interface":"front","average":{"1min":23.844000000000001,"5min":23.138000000000002,"15min":28.664000000000001},"min":{"1min":13.585000000000001,"5min":13.432,"15min":10.176},"max":{"1min":128.422,"5min":128.422,"15min":216.917},"last":21.239999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575127,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82560,"kb_used_data":6968,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94289280,"statfs":{"total":96636764160,"available":96552222720,"internally_reserved":0,"allocated":7135232,"data_stored":4075523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:56:18 2023","interfaces":[{"interface":"back","average":{"1min":15.638999999999999,"5min":12.65,"15min":20.079999999999998},"min":{"1min":4.9409999999999998,"5min":4.9039999999999999,"15min":4.2409999999999997},"max":{"1min":114.304,"5min":130.309,"15min":1086.095},"last":4.8289999999999997},{"interface":"front","average":{"1min":15.129,"5min":16.271999999999998,"15min":18.593},"min":{"1min":5.2149999999999999,"5min":4.0209999999999999,"15min":4.0209999999999999},"max":{"1min":80.692999999999998,"5min":141.636,"15min":403.21899999999999},"last":14.525}]},{"osd":1,"last update":"Thu Dec 7 07:55:36 2023","interfaces":[{"interface":"back","average":{"1min":20.469000000000001,"5min":15.369,"15min":17.274999999999999},"min":{"1min":6.234,"5min":5.4050000000000002,"15min":4.819},"max":{"1min":130.52500000000001,"5min":130.52500000000001,"15min":310.18299999999999},"last":11.724},{"interface":"front","average":{"1min":23.145,"5min":17.341000000000001,"15min":20.655000000000001},"min":{"1min":7.8479999999999999,"5min":6.4640000000000004,"15min":4.9290000000000003},"max":{"1min":113.23099999999999,"5min":113.23099999999999,"15min":263.452},"last":8.3780000000000001}]},{"osd":2,"last update":"Thu Dec 7 07:56:18 2023","interfaces":[{"interface":"back","average":{"1min":23.103000000000002,"5min":19.741,"15min":19.963999999999999},"min":{"1min":7.6900000000000004,"5min":6.149,"15min":6.149},"max":{"1min":169.61799999999999,"5min":169.61799999999999,"15min":248.95699999999999},"last":16.297000000000001},{"interface":"front","average":{"1min":20.579000000000001,"5min":18.251000000000001,"15min":18.809999999999999},"min":{"1min":8.2690000000000001,"5min":7.8360000000000003,"15min":6.4059999999999997},"max":{"1min":146.29599999999999,"5min":148.40100000000001,"15min":300.988},"last":12.6}]},{"osd":3,"last update":"Thu Dec 7 07:56:18 2023","interfaces":[{"interface":"back","average":{"1min":23.004999999999999,"5min":21.687999999999999,"15min":22.922000000000001},"min":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":6.9809999999999999},"max":{"1min":135.92400000000001,"5min":143.53899999999999,"15min":305.74900000000002},"last":17.106000000000002},{"interface":"front","average":{"1min":29.059000000000001,"5min":25.245999999999999,"15min":25.593},"min":{"1min":11.331,"5min":10.500999999999999,"15min":10.500999999999999},"max":{"1min":176.00700000000001,"5min":176.00700000000001,"15min":326.85899999999998},"last":17.869}]},{"osd":4,"last update":"Thu Dec 7 07:56:18 2023","interfaces":[{"interface":"back","average":{"1min":22.367999999999999,"5min":21.465,"15min":21.876000000000001},"min":{"1min":7.3150000000000004,"5min":6.5899999999999999,"15min":6.5899999999999999},"max":{"1min":131.65199999999999,"5min":136.691,"15min":311.52100000000002},"last":13.646000000000001},{"interface":"front","average":{"1min":22.527000000000001,"5min":23.443999999999999,"15min":22.004999999999999},"min":{"1min":10.131,"5min":8.7840000000000007,"15min":6.6929999999999996},"max":{"1min":100.245,"5min":146.81,"15min":313.42500000000001},"last":18.762}]},{"osd":6,"last update":"Thu Dec 7 07:56:18 2023","interfaces":[{"interface":"back","average":{"1min":46.314999999999998,"5min":28.998000000000001,"15min":30.344999999999999},"min":{"1min":10.053000000000001,"5min":9.1029999999999998,"15min":8.9280000000000008},"max":{"1min":443.43400000000003,"5min":443.43400000000003,"15min":443.43400000000003},"last":15.362},{"interface":"front","average":{"1min":25.143000000000001,"5min":26.384,"15min":28.170000000000002},"min":{"1min":11.657,"5min":10.384,"15min":6.3630000000000004},"max":{"1min":111.637,"5min":192.81800000000001,"15min":365.91899999999998},"last":19.562000000000001}]},{"osd":7,"last update":"Thu Dec 7 07:56:18 2023","interfaces":[{"interface":"back","average":{"1min":20.859000000000002,"5min":19.457000000000001,"15min":19.93},"min":{"1min":9.4350000000000005,"5min":6.6100000000000003,"15min":6.484},"max":{"1min":118.872,"5min":203.636,"15min":266.56299999999999},"last":9.0199999999999996},{"interface":"front","average":{"1min":24.227,"5min":23.815000000000001,"15min":25.748000000000001},"min":{"1min":12.329000000000001,"5min":8.4990000000000006,"15min":8.4990000000000006},"max":{"1min":121.066,"5min":204.45500000000001,"15min":300.18200000000002},"last":10.319000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575126,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59960,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311880,"statfs":{"total":96636764160,"available":96575365120,"internally_reserved":0,"allocated":6733824,"data_stored":3677683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 7 07:56:17 2023","interfaces":[{"interface":"back","average":{"1min":18.640999999999998,"5min":13.462999999999999,"15min":16.738},"min":{"1min":5.9669999999999996,"5min":4.5499999999999998,"15min":4.3230000000000004},"max":{"1min":127.751,"5min":127.751,"15min":301.959},"last":10.712},{"interface":"front","average":{"1min":16.446999999999999,"5min":15.669,"15min":20.149999999999999},"min":{"1min":6.7389999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"max":{"1min":56.146999999999998,"5min":110.508,"15min":363.75099999999998},"last":8.4710000000000001}]},{"osd":1,"last update":"Thu Dec 7 07:56:17 2023","interfaces":[{"interface":"back","average":{"1min":11.916,"5min":15.029999999999999,"15min":19.190999999999999},"min":{"1min":6.7089999999999996,"5min":6.4900000000000002,"15min":5.577},"max":{"1min":30.616,"5min":121.346,"15min":219.19300000000001},"last":11.44},{"interface":"front","average":{"1min":13.555,"5min":16.158000000000001,"15min":19.606999999999999},"min":{"1min":7.4320000000000004,"5min":6.3449999999999998,"15min":6.3209999999999997},"max":{"1min":27.91,"5min":111.107,"15min":158.66300000000001},"last":9.9459999999999997}]},{"osd":2,"last update":"Thu Dec 7 07:56:17 2023","interfaces":[{"interface":"back","average":{"1min":14.525,"5min":18.193999999999999,"15min":20.492000000000001},"min":{"1min":7.7160000000000002,"5min":7.7160000000000002,"15min":6.8710000000000004},"max":{"1min":27.123999999999999,"5min":168.87799999999999,"15min":176.452},"last":13.804},{"interface":"front","average":{"1min":16.048999999999999,"5min":18.643999999999998,"15min":22.015999999999998},"min":{"1min":8.8260000000000005,"5min":7.6020000000000003,"15min":7.6020000000000003},"max":{"1min":61.972999999999999,"5min":115.623,"15min":239.935},"last":15.917}]},{"osd":3,"last update":"Thu Dec 7 07:56:17 2023","interfaces":[{"interface":"back","average":{"1min":13.358000000000001,"5min":15.169,"15min":20.404},"min":{"1min":5.9359999999999999,"5min":5.9359999999999999,"15min":5.9359999999999999},"max":{"1min":46.473999999999997,"5min":173.18199999999999,"15min":237.16999999999999},"last":9.1910000000000007},{"interface":"front","average":{"1min":18.553999999999998,"5min":19.699999999999999,"15min":17.556999999999999},"min":{"1min":8.1950000000000003,"5min":6.9939999999999998,"15min":5.4240000000000004},"max":{"1min":60.091999999999999,"5min":183.75399999999999,"15min":183.75399999999999},"last":12.975}]},{"osd":5,"last update":"Thu Dec 7 07:56:17 2023","interfaces":[{"interface":"back","average":{"1min":13.276999999999999,"5min":15.428000000000001,"15min":18.545000000000002},"min":{"1min":6.5209999999999999,"5min":6.327,"15min":6.327},"max":{"1min":39.682000000000002,"5min":123.998,"15min":148.21000000000001},"last":17.422999999999998},{"interface":"front","average":{"1min":19.251999999999999,"5min":20.777999999999999,"15min":22.326000000000001},"min":{"1min":9.5030000000000001,"5min":9.5030000000000001,"15min":6.46},"max":{"1min":67.724999999999994,"5min":103.83799999999999,"15min":224.55500000000001},"last":18.901}]},{"osd":6,"last update":"Thu Dec 7 07:56:17 2023","interfaces":[{"interface":"back","average":{"1min":17.222000000000001,"5min":19.943999999999999,"15min":21.791},"min":{"1min":10.19,"5min":8.2129999999999992,"15min":7.8079999999999998},"max":{"1min":44.573999999999998,"5min":158.36099999999999,"15min":168.67599999999999},"last":15.212999999999999},{"interface":"front","average":{"1min":25.655999999999999,"5min":23.864999999999998,"15min":26.088000000000001},"min":{"1min":12.555999999999999,"5min":10.300000000000001,"15min":9.9299999999999997},"max":{"1min":135.63900000000001,"5min":196.833,"15min":243.40700000000001},"last":20.213999999999999}]},{"osd":7,"last update":"Thu Dec 7 07:56:17 2023","interfaces":[{"interface":"back","average":{"1min":21.925000000000001,"5min":22.786000000000001,"15min":25.748000000000001},"min":{"1min":13.161,"5min":9.9939999999999998,"15min":9.2319999999999993},"max":{"1min":83.831000000000003,"5min":146.798,"15min":225.375},"last":14.545999999999999},{"interface":"front","average":{"1min":17.567,"5min":17.356000000000002,"15min":18.628},"min":{"1min":7.4939999999999998,"5min":7.181,"15min":6.7110000000000003},"max":{"1min":76.287000000000006,"5min":85.081999999999994,"15min":167.53700000000001},"last":18.085999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T07:56:38.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.534+0000 7fbdb9b16640 10 monclient: get_monmap_and_config 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.534+0000 7fbdb9b16640 10 monclient: build_initial_monmap 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.534+0000 7fbdb9b16640 10 monclient: monmap: 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:56:38.535576+0000 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:56:38.535576+0000 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a-legacy 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2023-12-07T07:56:38.537 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-b 2023-12-07T07:56:38.538 INFO:teuthology.orchestra.run.smithi097.stderr:3: v2:172.21.15.97:3300/0 mon.noname-a 2023-12-07T07:56:38.538 INFO:teuthology.orchestra.run.smithi097.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2023-12-07T07:56:38.538 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 1 Processor -- start 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 1 -- start start 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient: init 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient: _reopen_session rank -1 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient(hunting): picked mon.noname-b con 0x7fbdb40a8a70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdb4142180 addr v1:172.21.15.97:6789/0 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbdb413e9c0 0x7fbdb413ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient(hunting): picked mon.noname-c con 0x7fbdb413e9c0 addr v2:172.21.15.154:3300/0 2023-12-07T07:56:38.540 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb40a9e70 con 0x7fbdb4142180 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb40aa9e0 con 0x7fbdb413e9c0 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb40aacc0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient(hunting): _renew_subs 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.535+0000 7fbdb9b16640 10 monclient(hunting): authenticate will time out at 2720.914165s 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb37fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb3fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbdb413e9c0 0x7fbdb413ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb37fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:41646/0 (socket says 172.21.15.97:41646) 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb37fe640 1 -- 172.21.15.97:0/4092407728 learned_addr learned my addr 172.21.15.97:0/4092407728 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb37fe640 10 monclient(hunting): get_auth_request con 0x7fbdb40a8a70 auth_method 0 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:38.541 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb3fff640 10 monclient(hunting): get_auth_request con 0x7fbdb413e9c0 auth_method 0 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb3fff640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.536+0000 7fbdb3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.537+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.537+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.537+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.537+0000 7fbdb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.537+0000 7fbdb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.537+0000 7fbdb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.538+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_done global_id 17729 payload 274 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.538+0000 7fbdb37fe640 1 -- 172.21.15.97:0/4092407728 >> v2:172.21.15.154:3300/0 conn(0x7fbdb413e9c0 msgr2=0x7fbdb413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.538+0000 7fbdb37fe640 1 --2- 172.21.15.97:0/4092407728 >> v2:172.21.15.154:3300/0 conn(0x7fbdb413e9c0 0x7fbdb413ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.538+0000 7fbdb37fe640 1 -- 172.21.15.97:0/4092407728 >> v1:172.21.15.97:6789/0 conn(0x7fbdb4142180 legacy=0x7fbdb41425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.538+0000 7fbdb37fe640 10 monclient: _finish_hunting 0 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.538+0000 7fbdb37fe640 1 monclient: found mon.noname-b 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.538+0000 7fbdb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.539+0000 7fbdb37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.97:3301/0 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.539+0000 7fbdb37fe640 1 -- 172.21.15.97:0/4092407728 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb40aafa0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.543 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.540+0000 7fbdb37fe640 1 --2- 172.21.15.97:0/4092407728 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40a8e50 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009870 tx=0x7fbda402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af29b5c6fa162163 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:38.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 1 -- 172.21.15.97:0/4092407728 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4031640 con 0x7fbdb40a8a70 2023-12-07T07:56:38.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:38.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-07T07:56:38.547 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: dump: 2023-12-07T07:56:38.547 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:38.547 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:38.547 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: _finish_auth 0 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: _check_auth_tickets 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 1 -- 172.21.15.97:0/4092407728 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda40317a0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 1 -- 172.21.15.97:0/4092407728 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4031b30 con 0x7fbdb40a8a70 2023-12-07T07:56:38.548 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 5 monclient: authenticate success, global_id 17729 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 10 monclient: get_monmap_and_config success 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 4 set_mon_vals no callback set 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb27fc640 10 monclient: dump: 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:38.549 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 10 monclient: shutdown 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 1 -- 172.21.15.97:0/4092407728 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 msgr2=0x7fbdb40a8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/4092407728 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40a8e50 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009870 tx=0x7fbda402b8f0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 1 -- 172.21.15.97:0/4092407728 shutdown_connections 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/4092407728 >> v2:172.21.15.154:3300/0 conn(0x7fbdb413e9c0 0x7fbdb413ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/4092407728 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40a8e50 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 1 -- 172.21.15.97:0/4092407728 >> 172.21.15.97:0/4092407728 conn(0x7fbdb4061a90 msgr2=0x7fbdb40b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.544+0000 7fbdb9b16640 1 -- 172.21.15.97:0/4092407728 shutdown_connections 2023-12-07T07:56:38.550 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.545+0000 7fbdb9b16640 1 -- 172.21.15.97:0/4092407728 wait complete. 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.545+0000 7fbdb9b16640 10 monclient: build_initial_monmap 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.545+0000 7fbdb9b16640 10 monclient: monmap: 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2023-12-07T07:56:38.553 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.noname-a 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.noname-c 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.545+0000 7fbdb9b16640 1 librados: starting msgr at 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.545+0000 7fbdb9b16640 1 librados: starting objecter 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.545+0000 7fbdb9b16640 1 Processor -- start 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 -- start start 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 librados: setting wanted keys 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 librados: calling monclient init 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient: init 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient: _reopen_session rank -1 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-07T07:56:38.554 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): picked mon.noname-c con 0x7fbdb40a8a70 addr [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb413e9c0 0x7fbdb40ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): picked mon.noname-b con 0x7fbdb413e9c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdb4142180 0x7fbdb41d7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): picked mon.noname-a con 0x7fbdb4142180 addr [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41c8fd0 con 0x7fbdb4142180 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41d7800 con 0x7fbdb413e9c0 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): start opening mon connection 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41c9200 con 0x7fbdb40a8a70 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): _renew_subs 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.546+0000 7fbdb9b16640 10 monclient(hunting): authenticate will time out at 2720.925072s 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb37fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb413e9c0 0x7fbdb40ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb37fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:41658/0 (socket says 172.21.15.97:41658) 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb37fe640 1 -- 172.21.15.97:0/3820800218 learned_addr learned my addr 172.21.15.97:0/3820800218 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb37fe640 10 monclient(hunting): get_auth_request con 0x7fbdb40a8a70 auth_method 0 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb37fe640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb2ffd640 10 monclient(hunting): get_auth_request con 0x7fbdb413e9c0 auth_method 0 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.547+0000 7fbdb2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.548+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:38.556 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.548+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.548+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.548+0000 7fbdb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.548+0000 7fbdb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.548+0000 7fbdb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 10 monclient(hunting): handle_auth_done global_id 17732 payload 824 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 1 -- 172.21.15.97:0/3820800218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb413e9c0 msgr2=0x7fbdb40ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb413e9c0 0x7fbdb40ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 1 -- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdb4142180 msgr2=0x7fbdb41d7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdb4142180 0x7fbdb41d7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 10 monclient: _finish_hunting 0 2023-12-07T07:56:38.557 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 1 monclient: found mon.noname-c 2023-12-07T07:56:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-07T07:56:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.97:3301/0 2023-12-07T07:56:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.550+0000 7fbdb37fe640 1 -- 172.21.15.97:0/3820800218 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda40094d0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.553+0000 7fbdb37fe640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40b1010 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fbda40090b0 tx=0x7fbda40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.553+0000 7fbd9bfff640 10 client.?.objecter ms_handle_connect 0x7fbdb40a8a70 2023-12-07T07:56:38.558 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.553+0000 7fbd9bfff640 10 client.?.objecter resend_mon_ops 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda403a970 con 0x7fbdb40a8a70 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: dump: 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:38.560 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: _finish_auth 0 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: _check_auth_tickets 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda403aaf0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda403aea0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 5 monclient: authenticate success, global_id 17732 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 10 monclient: _renew_subs 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb40af630 con 0x7fbdb40a8a70 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-07T07:56:38.561 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbd9bfff640 10 monclient: dump: 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:fsid f040f620-331a-4806-a30a-a56e2cce06a5 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:created 2023-12-07T07:32:00.612581+0000 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 18 (reef) 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb1ffb640 4 set_mon_vals no callback set 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 10 client.17732.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-07T07:56:38.562 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 10 monclient: _renew_subs 2023-12-07T07:56:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:56:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb41d8500 con 0x7fbdb40a8a70 2023-12-07T07:56:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.557+0000 7fbdb9b16640 1 librados: init done 2023-12-07T07:56:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.558+0000 7fbdb9b16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.558+0000 7fbdb9b16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:38.563 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.558+0000 7fbdb9b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:56:38.564 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.558+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb4067ac0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.579 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.577+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbda402f050 con 0x7fbdb40a8a70 2023-12-07T07:56:38.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.577+0000 7fbd9bfff640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fbda006c820 0x7fbda006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-07T07:56:38.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.577+0000 7fbdb2ffd640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fbda006c820 0x7fbda006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-07T07:56:38.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.577+0000 7fbdb2ffd640 10 monclient: get_auth_request con 0x7fbda006c820 auth_method 0 2023-12-07T07:56:38.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.578+0000 7fbd9bfff640 10 client.17732.objecter ms_handle_connect 0x7fbda006c820 2023-12-07T07:56:38.580 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.578+0000 7fbdb2ffd640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fbda006c820 0x7fbda006ece0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40b32b0 tx=0x7fbda8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-07T07:56:38.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(920..920 src has 600..920) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fbda407a9f0 con 0x7fbdb40a8a70 2023-12-07T07:56:38.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 10 client.17732.objecter ms_dispatch 0x7fbdb4000b90 osd_map(920..920 src has 600..920) v4 2023-12-07T07:56:38.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 3 client.17732.objecter handle_osd_map got epochs [920,920] > 0 2023-12-07T07:56:38.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 3 client.17732.objecter handle_osd_map decoding full epoch 920 2023-12-07T07:56:38.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 20 client.17732.objecter dump_active .. 0 homeless 2023-12-07T07:56:38.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbda4076630 con 0x7fbdb40a8a70 2023-12-07T07:56:38.819 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-07T07:56:38.820 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:38.816+0000 7fbd9bfff640 10 monclient: _finish_command 1 = system:0 2023-12-07T07:56:38.951 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:05.400 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:38.953 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:05.402 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:38.973 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:05.421 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:39.004 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:05.453 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:39.006 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:05.444 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:39.006 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:05.455 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:39.008 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:05.446 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:39.024 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:05.461 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:39.024 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:05.474 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:39.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.033+0000 7fbdb9b16640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-07T07:56:39.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.033+0000 7fbdb9b16640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-07T07:56:39.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.033+0000 7fbdb9b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.97:3301/0 2023-12-07T07:56:39.036 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.033+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fbdb41d8750 con 0x7fbdb40a8a70 2023-12-07T07:56:39.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.359+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbda407a2f0 con 0x7fbdb40a8a70 2023-12-07T07:56:39.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.359+0000 7fbd9bfff640 10 monclient: handle_config config(1 keys) v1 2023-12-07T07:56:39.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.359+0000 7fbd9bfff640 1 -- 172.21.15.97:0/3820800218 <== mon.2 v2:172.21.15.97:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fbdb41d8750 con 0x7fbdb40a8a70 2023-12-07T07:56:39.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.359+0000 7fbd9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-07T07:56:39.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.359+0000 7fbd9bfff640 10 monclient: _finish_command 2 = system:0 2023-12-07T07:56:39.362 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.359+0000 7fbdb1ffb640 4 set_mon_vals no callback set 2023-12-07T07:56:39.363 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.359+0000 7fbdb1ffb640 10 set_mon_vals mon_health_to_clog = false 2023-12-07T07:56:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 10 librados: watch_flush enter 2023-12-07T07:56:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 10 librados: watch_flush exit 2023-12-07T07:56:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 20 client.17732.objecter shutdown clearing up homeless session... 2023-12-07T07:56:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 10 client.17732.objecter successfully canceled tick 2023-12-07T07:56:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fbda006c820 msgr2=0x7fbda006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:39.365 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fbda006c820 0x7fbda006ece0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40b32b0 tx=0x7fbda8009040 comp rx=0 tx=0).stop 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 10 monclient: shutdown 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 msgr2=0x7fbdb40b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40b1010 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fbda40090b0 tx=0x7fbda40046c0 comp rx=0 tx=0).stop 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 shutdown_connections 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:6800/3287182838,v1:172.21.15.97:6801/3287182838] conn(0x7fbda006c820 0x7fbda006ece0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.362+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbdb4142180 0x7fbdb41d7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.363+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb413e9c0 0x7fbdb40ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.363+0000 7fbdb9b16640 1 --2- 172.21.15.97:0/3820800218 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbdb40a8a70 0x7fbdb40b1010 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.363+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 >> 172.21.15.97:0/3820800218 conn(0x7fbdb4061a90 msgr2=0x7fbdb40b5450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.363+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 shutdown_connections 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.363+0000 7fbdb9b16640 1 -- 172.21.15.97:0/3820800218 wait complete. 2023-12-07T07:56:39.366 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-07T07:56:39.363+0000 7fbdb9b16640 1 librados: shutdown 2023-12-07T07:56:39.380 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-07T07:56:39.381 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-07T07:56:39.381 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-07T07:56:39.381 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T07:56:39.397 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-12-07T07:56:39.383+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 87031) UID: 0 2023-12-07T07:56:39.397 INFO:tasks.ceph.osd.0.smithi097.stderr:2023-12-07T07:56:39.384+0000 b47e640 -1 osd.0 920 *** Got signal Terminated *** 2023-12-07T07:56:46.378 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:12.826 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.381 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:12.830 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.395 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:12.844 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.425 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:12.863 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.426 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:12.864 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.429 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:12.878 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.430 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:12.879 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.439 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:12.877 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:46.444 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:12.893 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.297 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:22.746 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.300 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:22.749 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.310 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:22.759 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.344 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:22.792 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.345 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:22.783 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.346 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:22.796 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.347 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:22.785 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.358 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:22.807 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:56.359 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:22.797 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:56:57.686 INFO:tasks.ceph.osd.0:Stopped 2023-12-07T07:56:57.686 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-07T07:56:57.686 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T07:56:57.698 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-12-07T07:56:57.688+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 87046) UID: 0 2023-12-07T07:56:57.698 INFO:tasks.ceph.osd.1.smithi097.stderr:2023-12-07T07:56:57.689+0000 b47e640 -1 osd.1 924 *** Got signal Terminated *** 2023-12-07T07:57:06.329 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:32.777 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.330 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:32.779 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.344 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:32.793 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.379 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:32.817 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.381 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:32.818 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.393 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:32.831 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.396 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:32.845 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.397 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:32.847 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:06.410 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:32.859 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.346 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:42.795 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.350 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:42.799 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.362 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:42.810 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.394 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:42.843 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.397 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:42.835 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.398 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:42.847 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.399 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:42.837 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.411 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:42.859 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:16.412 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:42.849 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:22.093 INFO:tasks.ceph.osd.1:Stopped 2023-12-07T07:57:22.094 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-07T07:57:22.094 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T07:57:22.106 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:57:22.096+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 87050) UID: 0 2023-12-07T07:57:22.106 INFO:tasks.ceph.osd.2.smithi097.stderr:2023-12-07T07:57:22.097+0000 b47e640 -1 osd.2 926 *** Got signal Terminated *** 2023-12-07T07:57:22.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-07T07:57:26.378 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:52.827 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.380 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:52.829 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.394 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:24:52.843 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.426 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:52.875 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.427 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:52.877 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.430 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:52.868 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.432 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:52.870 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.445 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:24:52.883 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:26.448 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:24:52.897 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:28.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-07T07:57:36.396 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:02.845 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.400 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:02.849 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.412 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:02.861 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.445 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:02.894 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.449 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:02.886 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.451 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:02.900 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.452 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:02.890 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.462 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:02.911 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:36.464 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:02.902 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.424 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:12.872 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.426 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:12.875 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.432 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:12.870 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.435 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:12.873 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.443 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:12.891 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.449 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:12.887 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.471 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:12.920 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.474 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:12.923 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:46.486 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:12.935 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.483 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:22.931 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.485 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:22.934 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.530 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:22.967 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.532 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:22.981 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.533 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:22.971 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.536 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:22.985 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.541 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:22.990 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.567 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:23.005 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:57:56.576 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:23.025 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.424 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:27.872 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.473 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:27.921 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.488 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:27.936 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.519 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:27.957 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.521 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:27.970 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.522 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:27.960 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.523 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:27.972 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.536 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:27.985 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:01.536 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:27.974 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.463 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:42.911 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.512 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:42.961 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.513 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:42.951 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.517 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:42.955 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.530 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:42.979 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.534 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:42.972 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.562 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:43.011 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.564 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:43.013 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:16.580 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:43.028 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.522 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:47.971 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.526 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:47.976 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.538 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:25:47.987 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.577 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:48.014 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.577 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:48.026 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.578 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:48.027 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.579 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:48.017 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.588 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:25:48.038 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:21.590 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:25:48.028 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.564 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:03.013 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.566 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:03.015 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.585 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:03.034 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.610 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:03.048 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.613 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:03.051 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.615 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:03.064 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.621 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:03.070 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.628 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:03.066 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:36.636 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:03.085 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.544 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:12.992 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.592 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:13.041 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.612 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:13.061 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.637 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:13.074 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.641 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:13.079 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.641 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:13.090 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.644 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:13.093 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.653 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:13.091 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:46.655 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:13.105 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.614 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:23.063 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.620 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:23.069 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.636 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:23.084 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.665 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:23.103 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.666 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:23.115 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.668 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:23.106 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.669 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:23.119 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.681 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:23.119 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:58:56.682 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:23.131 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.648 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:33.096 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.649 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:33.098 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.662 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:33.111 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.691 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:33.140 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.694 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:33.143 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.694 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:33.132 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.698 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:33.136 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.707 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:33.156 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:06.711 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:33.148 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.664 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:43.112 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.668 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:43.117 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.687 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:43.136 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.717 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:43.154 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.719 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:43.157 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.732 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:43.170 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.738 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:43.187 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.740 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:43.189 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:16.751 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:43.200 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.691 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:53.139 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.698 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:53.147 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.711 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:26:53.160 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.740 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:53.189 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.742 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:53.191 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.749 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:53.185 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.749 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:53.187 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.754 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:26:53.203 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:26.762 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:26:53.200 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.716 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:03.165 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.723 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:03.172 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.738 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:03.187 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.776 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:03.214 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.778 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:03.227 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.778 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:03.216 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.781 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:03.230 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.791 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:03.229 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:36.794 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:03.243 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.842 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:09.291 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.846 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:09.295 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.851 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:09.288 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.856 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:09.305 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.857 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:09.306 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.894 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:09.332 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.907 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:09.345 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.923 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:09.372 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:42.927 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:09.377 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.758 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:13.206 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.762 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:13.211 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.777 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:13.226 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.813 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:13.251 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.816 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:13.254 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.827 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:13.265 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.830 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:13.279 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.833 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:13.283 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:46.844 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:13.293 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.749 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:23.197 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.798 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:23.247 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.802 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:23.240 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.806 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:23.244 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.814 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:23.263 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.820 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:23.258 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.846 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:23.295 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.849 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:23.298 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T07:59:56.865 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:23.314 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.760 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:33.208 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.813 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:33.262 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.825 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:33.274 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.859 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:33.308 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.861 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:33.299 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.864 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:33.302 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.867 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:33.316 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.877 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:33.314 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:06.879 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:33.328 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.837 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:43.286 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.839 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:43.288 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.856 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:43.305 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.890 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:43.328 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.892 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:43.330 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.895 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:43.344 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.903 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:43.341 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.903 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:43.352 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:16.915 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:43.364 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:25.141 INFO:tasks.ceph.osd.2:Stopped 2023-12-07T08:00:25.141 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-07T08:00:25.141 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:00:25.153 INFO:tasks.ceph.osd.3.smithi097.stderr:2023-12-07T08:00:25.144+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 87054) UID: 0 2023-12-07T08:00:25.153 INFO:tasks.ceph.osd.3.smithi097.stderr:2023-12-07T08:00:25.145+0000 b47e640 -1 osd.3 928 *** Got signal Terminated *** 2023-12-07T08:00:25.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-07T08:00:26.986 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:53.435 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:26.989 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:53.438 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:27.017 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:27:53.466 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:27.038 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:53.487 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:27.039 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:53.488 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:27.041 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:53.479 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:27.042 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:53.480 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:27.054 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:27:53.492 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:27.057 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:27:53.506 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:31.243 INFO:tasks.ceph.osd.3:Stopped 2023-12-07T08:00:31.244 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-07T08:00:31.244 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:00:31.256 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-12-07T08:00:31.246+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86717) UID: 0 2023-12-07T08:00:31.257 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-12-07T08:00:31.247+0000 b47e640 -1 osd.4 930 *** Got signal Terminated *** 2023-12-07T08:00:31.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-07T08:00:31.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-07T08:00:36.997 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:03.445 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.002 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:03.451 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.017 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:03.466 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.044 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:03.482 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.045 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:03.483 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.048 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:03.497 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.052 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:03.502 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.061 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:03.499 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.068 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:03.517 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:37.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-12-07T08:00:46.939 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:13.388 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:46.955 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:13.404 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:46.972 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:13.420 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:46.985 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:13.434 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:46.987 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:13.436 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:46.988 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:13.426 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:46.989 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:13.427 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:47.001 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:13.438 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:47.002 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:13.451 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:56.980 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:23.428 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:56.982 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:23.431 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:57.014 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:23.463 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:57.030 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:23.468 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:57.031 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:23.469 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:57.033 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:23.482 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:57.035 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:23.484 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:57.043 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:23.481 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:00:57.050 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:23.499 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.010 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:33.457 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.010 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:33.459 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.023 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:33.472 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.060 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:33.498 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.061 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:33.499 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.069 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:33.518 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.071 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:33.520 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.074 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:33.511 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.083 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:33.532 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:07.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-12-07T08:01:07.853 INFO:tasks.ceph.osd.4:Stopped 2023-12-07T08:01:07.853 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-07T08:01:07.853 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:01:07.868 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-12-07T08:01:07.854+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86715) UID: 0 2023-12-07T08:01:07.868 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-12-07T08:01:07.855+0000 b47e640 -1 osd.5 934 *** Got signal Terminated *** 2023-12-07T08:01:12.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-12-07T08:01:12.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-12-07T08:01:13.956 INFO:tasks.ceph.osd.5:Stopped 2023-12-07T08:01:13.956 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-07T08:01:13.957 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:01:13.970 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-12-07T08:01:13.959+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86718) UID: 0 2023-12-07T08:01:13.973 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-12-07T08:01:13.960+0000 b47e640 -1 osd.6 936 *** Got signal Terminated *** 2023-12-07T08:01:17.041 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:43.490 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.243 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:43.492 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.243 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:43.506 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.243 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:43.523 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.244 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:43.526 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.244 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:43.540 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.244 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:43.541 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.244 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:43.542 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:17.244 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:43.556 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:18.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-12-07T08:01:18.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-12-07T08:01:20.059 INFO:tasks.ceph.osd.6:Stopped 2023-12-07T08:01:20.059 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-07T08:01:20.059 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:01:20.072 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T08:01:20.062+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86721) UID: 0 2023-12-07T08:01:20.075 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-12-07T08:01:20.063+0000 b47e640 -1 osd.7 936 *** Got signal Terminated *** 2023-12-07T08:01:22.049 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:48.498 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.055 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:48.504 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.085 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:48.534 86745== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.101 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:48.539 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.103 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:48.553 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.106 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:48.542 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.106 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:48.554 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.114 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:28:48.552 86180== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:22.116 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:28:48.565 86749== Warning: unimplemented fcntl command: 1036 2023-12-07T08:01:23.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-07T08:01:23.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-12-07T08:01:26.161 INFO:tasks.ceph.osd.7:Stopped 2023-12-07T08:01:26.161 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-07T08:01:26.161 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-07T08:01:26.161 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:01:26.162 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T08:01:26.160+0000 7f14195c0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 86754) UID: 0 2023-12-07T08:01:26.162 INFO:tasks.ceph.mgr.y.smithi097.stderr:2023-12-07T08:01:26.160+0000 7f14195c0640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-07T08:01:26.190 INFO:tasks.ceph.mgr.y:Stopped 2023-12-07T08:01:26.190 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-07T08:01:26.190 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:01:26.217 INFO:tasks.ceph.mgr.x:Stopped 2023-12-07T08:01:26.217 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-07T08:01:26.217 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-07T08:01:26.217 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:01:26.245 INFO:tasks.ceph.mon.a.smithi097.stderr:2023-12-07T08:01:26.232+0000 9f57640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86714) UID: 0 2023-12-07T08:01:26.245 INFO:tasks.ceph.mon.a.smithi097.stderr:2023-12-07T08:01:26.233+0000 9f57640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-07T08:01:27.021 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:53.470 86745== 2023-12-07T08:01:27.021 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:28:53.470 86745== Exit program on first error (--exit-on-first-error=yes) 2023-12-07T08:01:27.039 INFO:tasks.ceph.mon.a.smithi097.stderr:daemon-helper: command failed with exit status 42 2023-12-07T08:01:29.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-07T08:01:29.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-07T08:01:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-07T08:01:32.320 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-07T08:01:32.321 INFO:tasks.ceph.mon.a:Stopped 2023-12-07T08:01:32.321 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-07T08:01:32.322 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:01:32.336 INFO:tasks.ceph.mon.c.smithi097.stderr:2023-12-07T08:01:32.323+0000 9f57640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86716) UID: 0 2023-12-07T08:01:32.337 INFO:tasks.ceph.mon.c.smithi097.stderr:2023-12-07T08:01:32.325+0000 9f57640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-07T08:01:34.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-07T08:02:31.338 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:29:57.787 86749== 2023-12-07T08:02:31.339 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:29:57.787 86749== Exit program on first error (--exit-on-first-error=yes) 2023-12-07T08:02:31.354 INFO:tasks.ceph.mon.c.smithi097.stderr:daemon-helper: command failed with exit status 42 2023-12-07T08:02:31.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-07T08:02:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-07T08:02:33.338 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-07T08:02:33.339 INFO:tasks.ceph.mon.c:Stopped 2023-12-07T08:02:33.339 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-07T08:02:33.339 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T08:02:33.354 INFO:tasks.ceph.mon.b.smithi154.stderr:2023-12-07T08:02:33.342+0000 9f57640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86176) UID: 0 2023-12-07T08:02:33.355 INFO:tasks.ceph.mon.b.smithi154.stderr:2023-12-07T08:02:33.343+0000 9f57640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-12-07T08:02:34.117 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:30:00.555 86180== 2023-12-07T08:02:34.118 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:30:00.555 86180== Exit program on first error (--exit-on-first-error=yes) 2023-12-07T08:02:34.132 INFO:tasks.ceph.mon.b.smithi154.stderr:daemon-helper: command failed with exit status 42 2023-12-07T08:02:36.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-12-07T08:02:36.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-07T08:02:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-07T08:02:39.442 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-07T08:02:39.442 INFO:tasks.ceph.mon.b:Stopped 2023-12-07T08:02:39.442 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-07T08:02:39.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-12-07T08:02:39.484 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T08:02:39.484 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-07T08:02:39.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T08:02:39.610 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-07T08:02:39.692 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T08:02:39.692 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-07T08:02:39.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T08:02:39.741 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-07T08:02:39.817 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T08:02:39.817 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-07T08:02:40.017 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T08:02:40.017 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-07T08:02:40.099 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T08:02:40.100 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-07T08:02:40.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T08:02:40.156 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-07T08:02:40.212 INFO:tasks.ceph:Archiving mon data... 2023-12-07T08:02:40.214 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/data/mon.a.tgz 2023-12-07T08:02:40.215 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2023-12-07T08:02:40.232 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.P29ky7v57i 2023-12-07T08:02:40.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.P29ky7v57i 2023-12-07T08:02:40.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.P29ky7v57i 2023-12-07T08:02:40.486 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.P29ky7v57i is 503KB 2023-12-07T08:02:40.588 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.P29ky7v57i 2023-12-07T08:02:40.602 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/data/mon.c.tgz 2023-12-07T08:02:40.603 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2023-12-07T08:02:40.657 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.iZATuHy2SX 2023-12-07T08:02:40.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.iZATuHy2SX 2023-12-07T08:02:40.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.iZATuHy2SX 2023-12-07T08:02:40.897 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.iZATuHy2SX is 503KB 2023-12-07T08:02:40.996 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.iZATuHy2SX 2023-12-07T08:02:41.010 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/data/mon.b.tgz 2023-12-07T08:02:41.010 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2023-12-07T08:02:41.025 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.SL85OvkgLC 2023-12-07T08:02:41.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.SL85OvkgLC 2023-12-07T08:02:41.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.SL85OvkgLC 2023-12-07T08:02:41.285 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.SL85OvkgLC is 504KB 2023-12-07T08:02:41.380 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.SL85OvkgLC 2023-12-07T08:02:41.395 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-07T08:02:41.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-07T08:02:41.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-07T08:02:41.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-12-07T08:02:41.776 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-07T08:02:42.043 INFO:tasks.ceph:Archiving crash dumps... 2023-12-07T08:02:42.045 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/remote/smithi097/crash 2023-12-07T08:02:42.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-07T08:02:42.073 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/remote/smithi154/crash 2023-12-07T08:02:42.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-07T08:02:42.100 INFO:tasks.ceph:Compressing logs... 2023-12-07T08:02:42.100 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-07T08:02:42.118 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-07T08:02:42.191 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-07T08:02:42.191 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86573.log 2023-12-07T08:02:42.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86624.log 2023-12-07T08:02:42.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.86573.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-12-07T08:02:42.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.86624.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-12-07T08:02:42.191 INFO:teuthology.orchestra.run.smithi154.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86573.log.gz 2023-12-07T08:02:42.191 INFO:teuthology.orchestra.run.smithi154.stderr:gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.86624.log.gz --verbose -- 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr: /var/log/ceph/valgrind/osd.6.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/osd.7.log: /var/log/ceph/valgrind/osd.5.log: gzip 64.0% -5 --verbose -- replaced with /var/log/ceph/valgrind/osd.7.log.gz -- 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr: /var/log/ceph/valgrind/mon.b.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86333.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86533.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.86333.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.86533.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86333.log.gz 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-07T08:02:42.192 INFO:teuthology.orchestra.run.smithi154.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86533.log.gz 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86413.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.86413.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86413.log.gz 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86453.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.b.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86493.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.86453.log: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86453.log.gz 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.86373.log 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.86493.log: /var/log/ceph/ceph-client.admin.86373.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86493.log.gz 2023-12-07T08:02:42.193 INFO:teuthology.orchestra.run.smithi154.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.86373.log.gz 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93295.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91696.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92966.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92950.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92966.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91696.log.gz 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159446.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158608.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.159446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159154.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158608.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159446.log.gz 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.158734.log 2023-12-07T08:02:42.201 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92587.log 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158608.log.gz 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158734.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.159154.log.gz 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.91159.log 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92587.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112582.log 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158734.log.gz 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111713.log 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158566.log 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91159.log.gz/var/log/ceph/ceph-client.admin.111713.log: 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.112582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92939.log 2023-12-07T08:02:42.202 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158566.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.92950.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158566.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log/var/log/ceph/ceph-client.admin.92939.log: 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.93295.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr: 74.7%gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr: 91.1% 90.6% -- replaced with /var/log/ceph/ceph-client.admin.111713.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.112582.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2023-12-07T08:02:42.203 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.92966.log.gz 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93134.log 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92627.log 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158902.log 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158860.log 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158818.log 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91947.log 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.158902.log.gz 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158818.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.158860.log.gz 2023-12-07T08:02:42.204 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158818.log.gz 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158524.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86756.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92052.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91947.log.gz 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90975.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158524.log.gz 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92052.log: /var/log/ceph/ceph-client.admin.86756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92740.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0%/var/log/ceph/ceph-client.admin.90975.log: -- replaced with /var/log/ceph/ceph-client.admin.92052.log.gz 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91340.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90975.log.gz 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107862.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr: 91.4%/var/log/ceph/ceph-client.admin.91340.log: -- replaced with /var/log/ceph/ceph-client.admin.86756.log.gz 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93358.log 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91340.log.gz 2023-12-07T08:02:42.205 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.107862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90684.log 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93358.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.93358.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.93134.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118486.log 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92627.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.90684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91563.log 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.118486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112580.log 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr: 91.2% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90684.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107862.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92645.log 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.112580.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91563.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.159196.log 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138544.log 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159196.log: /var/log/ceph/ceph-client.admin.92645.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.118486.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159196.log.gz 2023-12-07T08:02:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.138544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91156.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.audit.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92645.log.gz 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 90.7% -- replaced with /var/log/ceph/ceph-client.admin.112580.log.gz 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr: --verbose -- /var/log/ceph/ceph-client.admin.159238.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159364.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159238.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91156.log.gz 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:gzip 87.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.93273.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159238.log.gz 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87289.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.138544.log.gz 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127281.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.159364.log.gz 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158692.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.127281.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87289.log.gz 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90844.log 2023-12-07T08:02:42.207 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158692.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.158692.log.gz 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158650.log 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.90844.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90844.log.gz 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.127281.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92032.log 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89060.log 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92032.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158650.log.gz 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92032.log.gz 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93354.log 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.89060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91384.log 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93354.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89060.log.gz 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93326.log 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91384.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91384.log.gz 2023-12-07T08:02:42.208 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93326.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.93354.log.gz 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91536.log 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.1.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.93326.log.gz 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80904.log 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91536.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91536.log.gz 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93089.log 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.tmp-client.admin.80904.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80904.log.gz 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92898.log 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93089.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127279.log 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92898.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.92898.log.gz 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158986.log 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93089.log.gz 2023-12-07T08:02:42.209 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.127279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158776.log 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158986.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158986.log.gz 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112581.log 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158776.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.158776.log.gz 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91142.log 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.112581.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.112581.log.gz 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.127279.log.gz 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127280.log 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138546.log 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91142.log.gz/var/log/ceph/ceph-client.admin.127280.log: 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118485.log 2023-12-07T08:02:42.210 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.138546.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.127280.log.gz 2023-12-07T08:02:42.211 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93048.log 2023-12-07T08:02:42.211 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.118485.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.138546.log.gz 2023-12-07T08:02:42.211 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-07T08:02:42.211 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93048.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.118485.log.gz 2023-12-07T08:02:42.211 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92609.log 2023-12-07T08:02:42.212 INFO:teuthology.orchestra.run.smithi154.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.3.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.93048.log.gz 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91816.log 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92609.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92609.log.gz 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91756.log 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91816.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91816.log.gz 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91161.log 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91756.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91756.log.gz 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.92587.log.gz 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159028.log 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92795.log 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91161.log.gz 2023-12-07T08:02:42.217 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91877.log 2023-12-07T08:02:42.218 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92795.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.159028.log.gz 2023-12-07T08:02:42.218 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91154.log 2023-12-07T08:02:42.219 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91877.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.93273.log.gz 2023-12-07T08:02:42.220 INFO:teuthology.orchestra.run.smithi097.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91877.log.gz 2023-12-07T08:02:42.220 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107863.log 2023-12-07T08:02:42.220 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158482.log 2023-12-07T08:02:42.220 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.107863.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91154.log.gz 2023-12-07T08:02:42.220 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138545.log 2023-12-07T08:02:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158482.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158482.log.gz 2023-12-07T08:02:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111714.log 2023-12-07T08:02:42.221 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.138545.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.107863.log.gz 2023-12-07T08:02:42.222 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93324.log 2023-12-07T08:02:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.111714.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.138545.log.gz 2023-12-07T08:02:42.224 INFO:teuthology.orchestra.run.smithi097.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.111714.log.gz 2023-12-07T08:02:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158944.log 2023-12-07T08:02:42.224 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91146.log 2023-12-07T08:02:42.225 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.158944.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.158944.log.gz 2023-12-07T08:02:42.225 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90888.log 2023-12-07T08:02:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91146.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91146.log.gz 2023-12-07T08:02:42.226 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91874.log 2023-12-07T08:02:42.227 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.90888.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90888.log.gz 2023-12-07T08:02:42.228 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91531.log 2023-12-07T08:02:42.230 INFO:teuthology.orchestra.run.smithi097.stderr: 90.9%/var/log/ceph/ceph-client.admin.91874.log: -- replaced with /var/log/ceph/ceph-client.admin.93324.log.gz 2023-12-07T08:02:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90734.log 2023-12-07T08:02:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91531.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91874.log.gz 2023-12-07T08:02:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92879.log 2023-12-07T08:02:42.230 INFO:teuthology.orchestra.run.smithi097.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91531.log.gz 2023-12-07T08:02:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.90734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90397.log 2023-12-07T08:02:42.231 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92879.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90734.log.gz 2023-12-07T08:02:42.231 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-07T08:02:42.232 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.90397.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90397.log.gz 2023-12-07T08:02:42.232 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92735.log 2023-12-07T08:02:42.236 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mgr.y.log: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.92939.log.gz 2023-12-07T08:02:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92132.log 2023-12-07T08:02:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92735.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.92879.log.gz 2023-12-07T08:02:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91466.log 2023-12-07T08:02:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92132.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92132.log.gz 2023-12-07T08:02:42.239 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91736.log 2023-12-07T08:02:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91466.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91466.log.gz 2023-12-07T08:02:42.241 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159322.log 2023-12-07T08:02:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91736.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91736.log.gz 2023-12-07T08:02:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose 91.6% -- /var/log/ceph/ceph-client.admin.92682.log -- replaced with /var/log/ceph/ceph-client.admin.92735.log.gz 2023-12-07T08:02:42.242 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T08:02:42.242 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87059.log 2023-12-07T08:02:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92682.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159322.log.gz 2023-12-07T08:02:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90930.log 2023-12-07T08:02:42.243 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87059.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87059.log.gz 2023-12-07T08:02:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92242.log 2023-12-07T08:02:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.90930.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90930.log.gz 2023-12-07T08:02:42.245 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93325.log 2023-12-07T08:02:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92242.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92242.log.gz 2023-12-07T08:02:42.246 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90795.log 2023-12-07T08:02:42.248 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93325.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.93325.log.gz 2023-12-07T08:02:42.248 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159070.log 2023-12-07T08:02:42.249 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.90795.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90795.log.gz 2023-12-07T08:02:42.249 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127282.log 2023-12-07T08:02:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159070.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159070.log.gz 2023-12-07T08:02:42.251 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93236.log 2023-12-07T08:02:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.127282.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.127282.log.gz 2023-12-07T08:02:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-07T08:02:42.264 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93236.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93236.log.gz 2023-12-07T08:02:42.264 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88132.log 2023-12-07T08:02:42.276 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92200.log 2023-12-07T08:02:42.277 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.88132.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88132.log.gz 2023-12-07T08:02:42.291 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91944.log 2023-12-07T08:02:42.292 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92200.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92200.log.gz 2023-12-07T08:02:42.299 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93059.log 2023-12-07T08:02:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91944.log: 92.8% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91944.log.gz 2023-12-07T08:02:42.305 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92682.log.gz 2023-12-07T08:02:42.306 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92756.log 2023-12-07T08:02:42.313 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93059.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.93059.log.gz 2023-12-07T08:02:42.315 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118487.log 2023-12-07T08:02:42.320 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92756.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92756.log.gz 2023-12-07T08:02:42.320 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92870.log 2023-12-07T08:02:42.324 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.118487.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.118487.log.gz 2023-12-07T08:02:42.325 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92112.log 2023-12-07T08:02:42.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92870.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-07T08:02:42.336 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92112.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92112.log.gz 2023-12-07T08:02:42.336 INFO:teuthology.orchestra.run.smithi097.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92870.log.gz 2023-12-07T08:02:42.336 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91380.log 2023-12-07T08:02:42.345 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.log: 92.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107861.log 2023-12-07T08:02:42.346 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91380.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.91380.log.gz 2023-12-07T08:02:42.346 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-12-07T08:02:42.346 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-07T08:02:42.350 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.107861.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.107861.log.gz 2023-12-07T08:02:42.350 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91150.log 2023-12-07T08:02:42.362 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91580.log 2023-12-07T08:02:42.363 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91150.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91150.log.gz 2023-12-07T08:02:42.373 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93168.log 2023-12-07T08:02:42.373 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91580.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91580.log.gz 2023-12-07T08:02:42.387 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93328.log 2023-12-07T08:02:42.389 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93168.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.93168.log.gz 2023-12-07T08:02:42.394 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93317.log 2023-12-07T08:02:42.406 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93328.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.93328.log.gz 2023-12-07T08:02:42.408 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127283.log 2023-12-07T08:02:42.409 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93317.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.93317.log.gz 2023-12-07T08:02:42.417 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159406.log 2023-12-07T08:02:42.428 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.127283.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.127283.log.gz 2023-12-07T08:02:42.428 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159280.log 2023-12-07T08:02:42.428 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159406.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.159406.log.gz 2023-12-07T08:02:42.441 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92284.log 2023-12-07T08:02:42.441 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159280.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.159280.log.gz 2023-12-07T08:02:42.449 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91153.log 2023-12-07T08:02:42.459 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92574.log 2023-12-07T08:02:42.463 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.91153.log: 87.4% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92284.log.gz 2023-12-07T08:02:42.470 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159112.log 2023-12-07T08:02:42.472 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.92574.log: -- replaced with /var/log/ceph/ceph-client.admin.91153.log.gz 2023-12-07T08:02:42.489 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93298.log 2023-12-07T08:02:42.490 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.159112.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.159112.log.gz 2023-12-07T08:02:42.504 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87161.log 2023-12-07T08:02:42.504 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.93298.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93298.log.gz 2023-12-07T08:02:42.520 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-07T08:02:42.521 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.87161.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87161.log.gz 2023-12-07T08:02:42.542 INFO:teuthology.orchestra.run.smithi097.stderr: 89.9%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.92574.log.gz 2023-12-07T08:02:42.595 INFO:teuthology.orchestra.run.smithi097.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-07T08:02:42.798 INFO:teuthology.orchestra.run.smithi097.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.92795.log.gz 2023-12-07T08:02:43.409 INFO:teuthology.orchestra.run.smithi154.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-07T08:02:43.512 INFO:teuthology.orchestra.run.smithi097.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92740.log.gz 2023-12-07T08:02:43.824 INFO:teuthology.orchestra.run.smithi097.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-07T08:02:45.069 INFO:teuthology.orchestra.run.smithi097.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-07T08:02:45.740 INFO:teuthology.orchestra.run.smithi154.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-07T08:02:46.072 INFO:teuthology.orchestra.run.smithi097.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-07T08:02:46.271 INFO:teuthology.orchestra.run.smithi154.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-07T08:02:46.289 INFO:teuthology.orchestra.run.smithi154.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-07T08:02:46.362 INFO:teuthology.orchestra.run.smithi154.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-07T08:02:46.363 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-12-07T08:02:46.363 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m4.203s 2023-12-07T08:02:46.363 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m16.684s 2023-12-07T08:02:46.364 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.552s 2023-12-07T08:02:46.374 INFO:teuthology.orchestra.run.smithi097.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-07T08:02:46.676 INFO:teuthology.orchestra.run.smithi097.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-07T08:02:46.826 INFO:teuthology.orchestra.run.smithi097.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-07T08:02:46.828 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-12-07T08:02:46.828 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m4.699s 2023-12-07T08:02:46.828 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m23.106s 2023-12-07T08:02:46.828 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.842s 2023-12-07T08:02:46.829 INFO:tasks.ceph:Archiving logs... 2023-12-07T08:02:46.829 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/remote/smithi097/log 2023-12-07T08:02:46.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-07T08:02:50.147 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/remote/smithi154/log 2023-12-07T08:02:50.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-07T08:02:53.676 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-07T08:02:53.677 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-07T08:02:53.686 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e068ebcf7ecc6503f24666fb6b152034d3fe1067/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-07T08:02:53.686 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-07T08:02:53.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-07T08:02:53.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-07T08:02:53.806 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-07T08:02:53.815 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-07T08:02:53.815 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T08:02:53.817 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T08:02:53.831 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2023-12-07T08:02:53.833 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2023-12-07T08:02:53.835 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-07T08:02:53.836 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2023-12-07T08:02:53.836 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +580us[ +571us] +/- 86ms 2023-12-07T08:02:53.836 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1379us[-1388us] +/- 53ms 2023-12-07T08:02:53.836 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +1349us[+1340us] +/- 79ms 2023-12-07T08:02:53.836 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T08:02:53.837 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-07T08:02:53.837 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2023-12-07T08:02:53.838 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +648us[ +664us] +/- 86ms 2023-12-07T08:02:53.838 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -1291us[-1274us] +/- 53ms 2023-12-07T08:02:53.838 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +1414us[+1430us] +/- 79ms 2023-12-07T08:02:53.838 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-07T08:02:53.839 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-07T08:02:53.848 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-07T08:02:53.849 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-07T08:02:53.859 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-07T08:02:53.975 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-07T08:02:54.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-07T08:02:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T08:02:54.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-07T08:02:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T08:02:54.140 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-07T08:02:54.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-07T08:02:54.157 INFO:teuthology.task.internal:Duration was 2745.396399 seconds 2023-12-07T08:02:54.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-07T08:02:54.166 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-07T08:02:54.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-07T08:02:54.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-07T08:02:54.210 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-07T08:02:54.224 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-07T08:02:54.501 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-07T08:02:54.501 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2023-12-07T08:02:54.502 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-07T08:02:54.520 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2023-12-07T08:02:54.521 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-07T08:02:54.546 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-07T08:02:54.547 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-07T08:02:54.563 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-07T08:02:54.616 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-07T08:02:54.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-07T08:02:54.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-07T08:02:54.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-07T08:02:54.763 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-07T08:02:54.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-07T08:02:54.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-07T08:02:54.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-07T08:02:54.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-07T08:02:54.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-07T08:02:54.855 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2023-12-07T08:02:54.882 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2023-12-07T08:02:54.908 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-07T08:02:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T08:02:54.926 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-07T08:02:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T08:02:54.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-07T08:02:54.972 INFO:teuthology.task.internal:Transferring archived files... 2023-12-07T08:02:54.973 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/remote/smithi097 2023-12-07T08:02:54.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-07T08:02:55.029 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571/remote/smithi154 2023-12-07T08:02:55.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-07T08:02:55.068 INFO:teuthology.task.internal:Removing archive directory... 2023-12-07T08:02:55.069 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-07T08:02:55.071 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-07T08:02:55.118 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-07T08:02:55.130 INFO:teuthology.task.internal:Not uploading archives. 2023-12-07T08:02:55.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-07T08:02:55.140 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-07T08:02:55.140 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-07T08:02:55.143 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-07T08:02:55.155 INFO:teuthology.orchestra.run.smithi097.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 7 08:02 /home/ubuntu/cephtest 2023-12-07T08:02:55.173 INFO:teuthology.orchestra.run.smithi154.stdout: 265256 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 7 08:02 /home/ubuntu/cephtest 2023-12-07T08:02:55.174 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-07T08:02:55.186 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-07T08:02:55.197 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate 2023-12-07T08:02:55.197 INFO:teuthology.nuke:Checking targets against current locks 2023-12-07T08:02:55.232 DEBUG:teuthology.nuke:shortname: smithi097 2023-12-07T08:02:55.232 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T08:02:55.276 DEBUG:teuthology.nuke:shortname: smithi154 2023-12-07T08:02:55.276 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T08:02:55.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 07:08:17.816140', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB4RfLkzGipXJRP4cCG467MgQNHYm2xufGqX5/6siE8P+xhpBRivA6vqSff7pu0DIsxm5y8b8cm90uALQU0ZS+o='} 2023-12-07T08:02:55.331 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_01:25:59-rados-wip-yuri8-testing-2023-12-06-1425-distro-default-smithi/7481571', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 07:08:17.817852', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeQXt1wvARrpJL3rnigyAmYqQUcE+qIhPN1th+Hbl2nIaVmXPX2zYdj7vZ1kMwMzNbFKgpVpWZumdUX91mLa0Q='} 2023-12-07T08:02:55.372 INFO:teuthology.orchestra.console.smithi097:Power off 2023-12-07T08:02:55.372 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T08:02:55.422 INFO:teuthology.orchestra.console.smithi154:Power off 2023-12-07T08:02:55.422 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T08:02:55.432 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2023-12-07T08:02:55.433 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T08:02:55.472 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2023-12-07T08:02:55.472 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T08:02:55.482 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2023-12-07T08:02:55.525 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-12-07T08:02:59.483 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T08:02:59.527 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T08:02:59.595 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2023-12-07T08:02:59.638 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-12-07T08:03:03.598 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T08:03:03.638 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T08:03:03.711 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2023-12-07T08:03:03.749 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-12-07T08:03:03.812 INFO:teuthology.orchestra.console.smithi097:Power off completed 2023-12-07T08:03:07.750 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T08:03:07.861 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2023-12-07T08:03:07.962 INFO:teuthology.orchestra.console.smithi154:Power off completed 2023-12-07T08:03:08.066 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 2745.3963992595673 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) allocate allocate' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-12-07T08:03:08.067 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-07T08:03:08.172 INFO:teuthology.run:FAIL